[jira] [Commented] (HDFS-5219) Add configuration keys for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770476#comment-13770476
]
Hadoop QA commented on HDFS-5219:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603756/HDFS-5219.005.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4994//console
This message is automatically generated.
Add configuration keys for WebHDFS
--
Key: HDFS-5219
URL: https://issues.apache.org/jira/browse/HDFS-5219
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-5219.000.patch, HDFS-5219.001.patch,
HDFS-5219.002.patch, HDFS-5219.003.patch, HDFS-5219.004.patch,
HDFS-5219.005.patch
Currently the WebHDFS client reuses the configuration of DFSClient to
construct its RetryPolicy. This is problematic because:
1. It makes the WebHDFS client depends on the DFSClient.
2. The default values of the RetryPolicy of DFSClient might be ill fit for
the WebHDFS client, which transfers all data using HTTP.
This JIRA proposes to introduce new configuration keys for WebHDFS to resolve
this problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5213) separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId
[
https://issues.apache.org/jira/browse/HDFS-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770478#comment-13770478
]
Andrew Wang commented on HDFS-5213:
---
Hey Colin,
Naming:
* Can we rename {{PathBasedCacheDirectiveWithId}} to something more general,
like {{PathBasedCacheDescriptor}}? Reminiscent of file descriptor.
* If {{PathBasedCacheDirectiveWithId}} is a client-side class, the javadoc
shouldn't be saying that it's an entry in the NameNode's PathBasedCache. That
sounds like a {{PathBasedCacheEntry}}.
* The {{PathBasedCacheEntry}} javadoc also needs to be improved
* {{getDirective()}} should be named {{getDirectiveWithId()}} (or
{{getDescriptor()}} if you use my suggestion).
* Exception text shouldn't talk about a {{PathBasedCache entry}} when it's
supposed to be a private namenode class. There's also no {{PathBasedCache}}
class, so confusing.
Nits/other:
* For implementing equals, how about comparing {{getClass()}} rather than
casting? This means it won't take subclasses, which is normally a good thing
(safe symmetric equals).
* I don't think any of this should be marked {{InterfaceAudience.Stable}} yet.
* extra {{mortbay.log}} import
Unrelated-but-related:
* I still want to see Paths, not Strings, for the DFS methods involving cache
directives. It's non-standard and weird for the reasons I mentioned before, and
DFS has error checking for being passed a URI for a different FileSystem.
separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId
--
Key: HDFS-5213
URL: https://issues.apache.org/jira/browse/HDFS-5213
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-5213-caching.001.patch
Since PathBasedCacheEntry is intended to be a private (implementation) class,
return PathBasedCacheDirectiveWithId from all public APIs instead. Some
other miscellaneous cleanups in the caching RPC stuff.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5219) Add configuration keys for WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5219: - Attachment: HDFS-5219.005.patch Add configuration keys for WebHDFS -- Key: HDFS-5219 URL: https://issues.apache.org/jira/browse/HDFS-5219 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-5219.000.patch, HDFS-5219.001.patch, HDFS-5219.002.patch, HDFS-5219.003.patch, HDFS-5219.004.patch, HDFS-5219.005.patch Currently the WebHDFS client reuses the configuration of DFSClient to construct its RetryPolicy. This is problematic because: 1. It makes the WebHDFS client depends on the DFSClient. 2. The default values of the RetryPolicy of DFSClient might be ill fit for the WebHDFS client, which transfers all data using HTTP. This JIRA proposes to introduce new configuration keys for WebHDFS to resolve this problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5219) Add configuration keys for WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5219: - Attachment: (was: HDFS-5219.005.patch) Add configuration keys for WebHDFS -- Key: HDFS-5219 URL: https://issues.apache.org/jira/browse/HDFS-5219 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-5219.000.patch, HDFS-5219.001.patch, HDFS-5219.002.patch, HDFS-5219.003.patch, HDFS-5219.004.patch, HDFS-5219.005.patch Currently the WebHDFS client reuses the configuration of DFSClient to construct its RetryPolicy. This is problematic because: 1. It makes the WebHDFS client depends on the DFSClient. 2. The default values of the RetryPolicy of DFSClient might be ill fit for the WebHDFS client, which transfers all data using HTTP. This JIRA proposes to introduce new configuration keys for WebHDFS to resolve this problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5212) Refactor RpcMessage and NFS3Response to support different types of authentication information
[ https://issues.apache.org/jira/browse/HDFS-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770483#comment-13770483 ] Hudson commented on HDFS-5212: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4433 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4433/]) HDFS-5212. Refactor RpcMessage and NFS3Response to support different types of authentication information. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1524298) * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/ACCESS3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/COMMIT3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/CREATE3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/FSINFO3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/FSSTAT3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/GETATTR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/LOOKUP3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/MKDIR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/NFS3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/PATHCONF3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READ3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READDIR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READDIRPLUS3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READLINK3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/REMOVE3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/RENAME3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/RMDIR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/SETATTR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/SYMLINK3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/VoidResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/WRITE3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcAcceptedReply.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcCall.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcDeniedReply.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcMessage.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcProgram.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcReply.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/security/Credentials.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/security/Verifier.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/PortmapRequest.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/PortmapResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/RpcProgramPortmap.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestFrameDecoder.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestRpcAcceptedReply.java *
[jira] [Updated] (HDFS-5199) Add more debug trace for NFS READ and WRITE
[ https://issues.apache.org/jira/browse/HDFS-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5199: - Priority: Trivial (was: Major) Add more debug trace for NFS READ and WRITE --- Key: HDFS-5199 URL: https://issues.apache.org/jira/browse/HDFS-5199 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Brandon Li Assignee: Brandon Li Priority: Trivial Fix For: 2.1.1-beta Attachments: HDFS-5199.2.patch, HDFS-5199.patch Before more sophisticated utility is added, the simple trace indicating start/end serving request can help debug errors and collect statistic information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5212) Refactor RpcMessage and NFS3Response to support different types of authentication information
[ https://issues.apache.org/jira/browse/HDFS-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5212: Resolution: Fixed Fix Version/s: 2.1.1-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the review Brandon! I've committed this to trunk, branch-2 and branch-2.1-beta. Refactor RpcMessage and NFS3Response to support different types of authentication information - Key: HDFS-5212 URL: https://issues.apache.org/jira/browse/HDFS-5212 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Jing Zhao Assignee: Jing Zhao Fix For: 2.1.1-beta Attachments: HDFS-5212.001.patch, HDFS-5212.002.patch Currently the authentication information is hard coded in RpcMessage (and its subclasses) and NFS3Response (and its subclasses) as AuthFlavor value and empty byte array. We need to refactor this part of code to support different types of authentication information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5066) Inode tree with snapshot information visualization
[ https://issues.apache.org/jira/browse/HDFS-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770513#comment-13770513 ] Jing Zhao commented on HDFS-5066: - Sorry for the delay, Binglin.. So in general I really like this feature. However, after some more thinking and some offline discussion with others, I think a tool that can only visualize several hundred nodes may not be very helpful in practice, where multiple million files/directories are stored in HDFS. It is more like a very useful tool for debugging unit test, in which the whole fsdirectory usually consists of that scale of inodes. In that case, unless we can make the tool continue the visualization based on the last time result, we may not want to add new ClientProtocol/DistributedFileSystem API for this feature. I.e., we may only want to keep the visualization functionality as a debugging feature in namenode. So do you have any idea to continue the visualization? Or any good use case that needs to add this functionality as a DistributedFileSystem API? Inode tree with snapshot information visualization --- Key: HDFS-5066 URL: https://issues.apache.org/jira/browse/HDFS-5066 Project: Hadoop HDFS Issue Type: Improvement Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor Attachments: HDFS-5066.v1.patch, HDFS-5066.v2.patch, visnap.png It would be nice to be able to visualize snapshot information, in order to ease the understanding of related data structures. We can generate graph from in memory inode links. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5219) Add configuration keys for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770562#comment-13770562
]
Hadoop QA commented on HDFS-5219:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603763/HDFS-5219.005.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4995//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4995//console
This message is automatically generated.
Add configuration keys for WebHDFS
--
Key: HDFS-5219
URL: https://issues.apache.org/jira/browse/HDFS-5219
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-5219.000.patch, HDFS-5219.001.patch,
HDFS-5219.002.patch, HDFS-5219.003.patch, HDFS-5219.004.patch,
HDFS-5219.005.patch
Currently the WebHDFS client reuses the configuration of DFSClient to
construct its RetryPolicy. This is problematic because:
1. It makes the WebHDFS client depends on the DFSClient.
2. The default values of the RetryPolicy of DFSClient might be ill fit for
the WebHDFS client, which transfers all data using HTTP.
This JIRA proposes to introduce new configuration keys for WebHDFS to resolve
this problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5031) BlockScanner scans the block multiple times and on restart scans everything
[
https://issues.apache.org/jira/browse/HDFS-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770619#comment-13770619
]
Vinay commented on HDFS-5031:
-
bq. I am still not convinced that the assignment to lastReadFile before the
call to readNext is correct. Is lastReadFile meant to store the file from which
the last line was read? If so then the call to readNext can change file, or did
I understand it wrong?
Here I agree that, {{readNext()}} will change the reference of {{file}}, but
{{next()}} will return the {{curLine}} which was read in the previous call of
{{readNext()}}, so since we are using the value of line before {{readNext()}}
in current call, we should also have the previous value of {{file}} for
{{lastReadFile}}. Otherwise, following problem will come.
# Consider {{RollingLogsImpl#next()}} call is expected to return the last but
one entry from {{dncp_block_verification.log.prev}}, during this time
{{RollingLogsImpl#readNext()}} would read the last entry and keep in {{line}}
# one more call to {{RollingLogsImpl#next()}}will return last entry read in
previous call, but this time {{readNext()}} will open
{{dncp_block_verification.log.cur}} and change {{file}} to
{{dncp_block_verification.log.cur}}.
# Now in {{BlockPoolSliceScanner#assignInitialVerificationTimes()}} while
processing the last entry from prev dncp log, if {{logIterator.isPrevious()}}
is called, then it will return false as the {{file}} have reference to current
verification log. Hence this entry will not be appended to current verification
log and block will be re-scanned after next roll.
{code:java}if (logIterator.isPrevious()) {
// write the log entry to current file
// so that the entry is preserved for later runs.
verificationLog.append(entry.verificationTime, entry.genStamp,
entry.blockId);
}
{code}
But {{logIterator.isLastReadFromPrevious()}} will return the true in this case
and no entry from prev dncp log will be missed.
BlockScanner scans the block multiple times and on restart scans everything
---
Key: HDFS-5031
URL: https://issues.apache.org/jira/browse/HDFS-5031
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Vinay
Assignee: Vinay
Attachments: HDFS-5031.patch, HDFS-5031.patch, HDFS-5031.patch
BlockScanner scans the block twice, also on restart of datanode scans
everything.
Steps:
1. Write blocks with interval of more than 5 seconds. write new block on
completion of scan for written block.
Each time datanode scans new block, it also scans, previous block which is
already scanned.
Now after restart, datanode scans all blocks again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5212) Refactor RpcMessage and NFS3Response to support different types of authentication information
[ https://issues.apache.org/jira/browse/HDFS-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770649#comment-13770649 ] Hudson commented on HDFS-5212: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #336 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/336/]) HDFS-5212. Refactor RpcMessage and NFS3Response to support different types of authentication information. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1524298) * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/ACCESS3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/COMMIT3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/CREATE3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/FSINFO3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/FSSTAT3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/GETATTR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/LOOKUP3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/MKDIR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/NFS3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/PATHCONF3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READ3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READDIR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READDIRPLUS3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READLINK3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/REMOVE3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/RENAME3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/RMDIR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/SETATTR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/SYMLINK3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/VoidResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/WRITE3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcAcceptedReply.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcCall.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcDeniedReply.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcMessage.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcProgram.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcReply.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/security/Credentials.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/security/Verifier.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/PortmapRequest.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/PortmapResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/RpcProgramPortmap.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestFrameDecoder.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestRpcAcceptedReply.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestRpcDeniedReply.java *
[jira] [Updated] (HDFS-4990) Block placement for storage types
[ https://issues.apache.org/jira/browse/HDFS-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4990: - Attachment: h4990_20130918b.patch Junping, thanks for taking a look. h4990_20130918b.patch: addresses Junping's comment. Block placement for storage types - Key: HDFS-4990 URL: https://issues.apache.org/jira/browse/HDFS-4990 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Suresh Srinivas Assignee: Tsz Wo (Nicholas), SZE Attachments: h4990_20130909.patch, h4990_20130916.patch, h4990_20130917b.patch, h4990_20130917c.patch, h4990_20130917.patch, h4990_20130918b.patch, h4990_20130918.patch Currently block location for writes are made based on: # Datanode load (number of transceivers) # Space left on datanode # Topology With storage abstraction, namenode must choose a storage instead of a datanode for block placement. It also needs to consider storage type, load on the storage etc. As an additional benefit, currently HDFS support heterogeneous nodes (nodes with different number of spindles etc.) poorly. This work should help solve that issue as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5212) Refactor RpcMessage and NFS3Response to support different types of authentication information
[ https://issues.apache.org/jira/browse/HDFS-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770676#comment-13770676 ] Hudson commented on HDFS-5212: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1526 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1526/]) HDFS-5212. Refactor RpcMessage and NFS3Response to support different types of authentication information. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1524298) * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/ACCESS3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/COMMIT3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/CREATE3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/FSINFO3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/FSSTAT3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/GETATTR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/LOOKUP3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/MKDIR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/NFS3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/PATHCONF3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READ3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READDIR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READDIRPLUS3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READLINK3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/REMOVE3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/RENAME3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/RMDIR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/SETATTR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/SYMLINK3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/VoidResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/WRITE3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcAcceptedReply.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcCall.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcDeniedReply.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcMessage.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcProgram.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcReply.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/security/Credentials.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/security/Verifier.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/PortmapRequest.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/PortmapResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/RpcProgramPortmap.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestFrameDecoder.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestRpcAcceptedReply.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestRpcDeniedReply.java *
[jira] [Commented] (HDFS-4990) Block placement for storage types
[ https://issues.apache.org/jira/browse/HDFS-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770746#comment-13770746 ] Junping Du commented on HDFS-4990: -- +1. Patch looks good to me. Thanks for addressing my comments, Nicholas! Block placement for storage types - Key: HDFS-4990 URL: https://issues.apache.org/jira/browse/HDFS-4990 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Suresh Srinivas Assignee: Tsz Wo (Nicholas), SZE Attachments: h4990_20130909.patch, h4990_20130916.patch, h4990_20130917b.patch, h4990_20130917c.patch, h4990_20130917.patch, h4990_20130918b.patch, h4990_20130918.patch Currently block location for writes are made based on: # Datanode load (number of transceivers) # Space left on datanode # Topology With storage abstraction, namenode must choose a storage instead of a datanode for block placement. It also needs to consider storage type, load on the storage etc. As an additional benefit, currently HDFS support heterogeneous nodes (nodes with different number of spindles etc.) poorly. This work should help solve that issue as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5212) Refactor RpcMessage and NFS3Response to support different types of authentication information
[ https://issues.apache.org/jira/browse/HDFS-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770760#comment-13770760 ] Hudson commented on HDFS-5212: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1552 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1552/]) HDFS-5212. Refactor RpcMessage and NFS3Response to support different types of authentication information. Contributed by Jing Zhao. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1524298) * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/mount/MountResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/ACCESS3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/COMMIT3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/CREATE3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/FSINFO3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/FSSTAT3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/GETATTR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/LOOKUP3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/MKDIR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/NFS3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/PATHCONF3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READ3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READDIR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READDIRPLUS3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/READLINK3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/REMOVE3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/RENAME3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/RMDIR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/SETATTR3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/SYMLINK3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/VoidResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/response/WRITE3Response.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcAcceptedReply.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcCall.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcDeniedReply.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcMessage.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcProgram.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcReply.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/security/Credentials.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/security/Verifier.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/PortmapRequest.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/PortmapResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/RpcProgramPortmap.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestFrameDecoder.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestRpcAcceptedReply.java *
[jira] [Commented] (HDFS-5066) Inode tree with snapshot information visualization
[ https://issues.apache.org/jira/browse/HDFS-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770825#comment-13770825 ] Binglin Chang commented on HDFS-5066: - Thanks for the comments Jing Zhao, your concern makes sense. I also hesitated to add new API in ClientProtocol, but for the running namenode service, there is few interfaces can be used to invoke the visualization, another way is through web servlet and add a config to only expose the servlet when config is enabled. If we don't expose this tool, it can only be used in debugging unit test. I made this tool mostly for developers to understand snapshot internal and debugging, visualization of millions files/directory does not seem possible and useful(it has way to much information). So I think the options left are expose web servlet or just left it internal and used by tests. What do you mean by make the tool continue the visualization based on the last time result? Do you mean a dynamic visualization updates along with fs changes? Inode tree with snapshot information visualization --- Key: HDFS-5066 URL: https://issues.apache.org/jira/browse/HDFS-5066 Project: Hadoop HDFS Issue Type: Improvement Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor Attachments: HDFS-5066.v1.patch, HDFS-5066.v2.patch, visnap.png It would be nice to be able to visualize snapshot information, in order to ease the understanding of related data structures. We can generate graph from in memory inode links. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4990) Block placement for storage types
[ https://issues.apache.org/jira/browse/HDFS-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-4990. -- Resolution: Fixed Fix Version/s: Heterogeneous Storage (HDFS-2832) Hadoop Flags: Reviewed Thanks Arpit and Junping for reviewing the patches. I have committed this. Block placement for storage types - Key: HDFS-4990 URL: https://issues.apache.org/jira/browse/HDFS-4990 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Suresh Srinivas Assignee: Tsz Wo (Nicholas), SZE Fix For: Heterogeneous Storage (HDFS-2832) Attachments: h4990_20130909.patch, h4990_20130916.patch, h4990_20130917b.patch, h4990_20130917c.patch, h4990_20130917.patch, h4990_20130918b.patch, h4990_20130918.patch Currently block location for writes are made based on: # Datanode load (number of transceivers) # Space left on datanode # Topology With storage abstraction, namenode must choose a storage instead of a datanode for block placement. It also needs to consider storage type, load on the storage etc. As an additional benefit, currently HDFS support heterogeneous nodes (nodes with different number of spindles etc.) poorly. This work should help solve that issue as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5222) Move block schedule information from DatanodeDescriptor to DatanodeStorageInfo
Tsz Wo (Nicholas), SZE created HDFS-5222: Summary: Move block schedule information from DatanodeDescriptor to DatanodeStorageInfo Key: HDFS-5222 URL: https://issues.apache.org/jira/browse/HDFS-5222 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE In HDFS-4990, the block placement target type was changed from DatanodeDescriptor to DatanodeStorageInfo. The block schedule information, such as the number of blocks scheduled for replication (i.e. getBlocksScheduled()), should be moved from DatanodeDescriptor to DatanodeStorageInfo. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5208) Only clear network location cache on specific nodes if invalid NetworkTopology happens
[
https://issues.apache.org/jira/browse/HDFS-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770910#comment-13770910
]
Junping Du commented on HDFS-5208:
--
Hi Colin, I think you are right that DatanodeID is created in DN heartbeat to
NN for registration and its hostName comes from conf of dfs.datanode.hostname
which can be any style but DNS name if this config is not setting.
However, following code in resolveNetworkLocation() called by
DatanodeManager.registerDatanode() make only IPs are cached through DN
registration. Isn't it?
{code}
if (dnsToSwitchMapping instanceof CachedDNSToSwitchMapping) {
names.add(node.getIpAddr());
} else {
names.add(node.getHostName());
}
{code}
Actually, now I am worrying about non-cached case, as even topology script can
resolve user-specified hostName to correct network location (rack) properly
and use it to register into networktopology tree. Later, it still need to
resolve topology based on nodes' IP (like in
DatanodeManager.sortLocatedBlocks()) which means script must contains both
user-specified hostName and IP for each node. IMO, This is really unnecessary
and confusing. Thoughts?
Only clear network location cache on specific nodes if invalid
NetworkTopology happens
--
Key: HDFS-5208
URL: https://issues.apache.org/jira/browse/HDFS-5208
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Junping Du
Assignee: Junping Du
Attachments: HDFS-5208-v1.patch
After HDFS-4521, once a DN is registered with invalid networktopology, all
cached rack info in DNSToSwitchMapping will be cleared. We should only clear
cache on specific nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5219) Add configuration keys for WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770981#comment-13770981 ] Jing Zhao commented on HDFS-5219: - +1. I will commit soon. Add configuration keys for WebHDFS -- Key: HDFS-5219 URL: https://issues.apache.org/jira/browse/HDFS-5219 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-5219.000.patch, HDFS-5219.001.patch, HDFS-5219.002.patch, HDFS-5219.003.patch, HDFS-5219.004.patch, HDFS-5219.005.patch Currently the WebHDFS client reuses the configuration of DFSClient to construct its RetryPolicy. This is problematic because: 1. It makes the WebHDFS client depends on the DFSClient. 2. The default values of the RetryPolicy of DFSClient might be ill fit for the WebHDFS client, which transfers all data using HTTP. This JIRA proposes to introduce new configuration keys for WebHDFS to resolve this problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5219) Add configuration keys for retry policy in WebHDFSFileSystem
[ https://issues.apache.org/jira/browse/HDFS-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5219: Summary: Add configuration keys for retry policy in WebHDFSFileSystem (was: Add configuration keys for WebHDFS) Add configuration keys for retry policy in WebHDFSFileSystem Key: HDFS-5219 URL: https://issues.apache.org/jira/browse/HDFS-5219 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-5219.000.patch, HDFS-5219.001.patch, HDFS-5219.002.patch, HDFS-5219.003.patch, HDFS-5219.004.patch, HDFS-5219.005.patch Currently the WebHDFS client reuses the configuration of DFSClient to construct its RetryPolicy. This is problematic because: 1. It makes the WebHDFS client depends on the DFSClient. 2. The default values of the RetryPolicy of DFSClient might be ill fit for the WebHDFS client, which transfers all data using HTTP. This JIRA proposes to introduce new configuration keys for WebHDFS to resolve this problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5219) Add configuration keys for retry policy in WebHDFSFileSystem
[ https://issues.apache.org/jira/browse/HDFS-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770995#comment-13770995 ] Hudson commented on HDFS-5219: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4435 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4435/]) HDFS-5219. Add configuration keys for retry policy in WebHDFSFileSystem. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1524498) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java Add configuration keys for retry policy in WebHDFSFileSystem Key: HDFS-5219 URL: https://issues.apache.org/jira/browse/HDFS-5219 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-5219.000.patch, HDFS-5219.001.patch, HDFS-5219.002.patch, HDFS-5219.003.patch, HDFS-5219.004.patch, HDFS-5219.005.patch Currently the WebHDFS client reuses the configuration of DFSClient to construct its RetryPolicy. This is problematic because: 1. It makes the WebHDFS client depends on the DFSClient. 2. The default values of the RetryPolicy of DFSClient might be ill fit for the WebHDFS client, which transfers all data using HTTP. This JIRA proposes to introduce new configuration keys for WebHDFS to resolve this problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5219) Add configuration keys for retry policy in WebHDFSFileSystem
[ https://issues.apache.org/jira/browse/HDFS-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5219: Resolution: Fixed Fix Version/s: 2.1.1-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this to trunk, branch-2 and branch-2.1-beta. Thanks [~wheat9]! Add configuration keys for retry policy in WebHDFSFileSystem Key: HDFS-5219 URL: https://issues.apache.org/jira/browse/HDFS-5219 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Reporter: Haohui Mai Assignee: Haohui Mai Fix For: 2.1.1-beta Attachments: HDFS-5219.000.patch, HDFS-5219.001.patch, HDFS-5219.002.patch, HDFS-5219.003.patch, HDFS-5219.004.patch, HDFS-5219.005.patch Currently the WebHDFS client reuses the configuration of DFSClient to construct its RetryPolicy. This is problematic because: 1. It makes the WebHDFS client depends on the DFSClient. 2. The default values of the RetryPolicy of DFSClient might be ill fit for the WebHDFS client, which transfers all data using HTTP. This JIRA proposes to introduce new configuration keys for WebHDFS to resolve this problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4971) Move IO operations out of locking in OpenFileCtx
[ https://issues.apache.org/jira/browse/HDFS-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4971: Attachment: HDFS-4971.004.patch Rebase the patch. Move IO operations out of locking in OpenFileCtx Key: HDFS-4971 URL: https://issues.apache.org/jira/browse/HDFS-4971 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-4971.000.patch, HDFS-4971.001.patch, HDFS-4971.002.patch, HDFS-4971.003.patch, HDFS-4971.004.patch Currently some IO operations (such as writing data to HDFS and dumping to local disk) in OpenFileCtx may hold a lock which can block processing incoming writing requests. This jira aims to optimize OpenFileCtx and move the IO operations out of the locking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5223) Allow edit log/fsimage format changes without changing layout version
Aaron T. Myers created HDFS-5223: Summary: Allow edit log/fsimage format changes without changing layout version Key: HDFS-5223 URL: https://issues.apache.org/jira/browse/HDFS-5223 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.1.1-beta Reporter: Aaron T. Myers Currently all HDFS on-disk formats are version by the single layout version. This means that even for changes which might be backward compatible, like the addition of a new edit log op code, we must go through the full `namenode -upgrade' process which requires coordination with DNs, etc. HDFS should support a lighter weight alternative. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5213) separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId
[
https://issues.apache.org/jira/browse/HDFS-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771051#comment-13771051
]
Colin Patrick McCabe commented on HDFS-5213:
bq. Can we rename PathBasedCacheDirectiveWithId to something more general, like
PathBasedCacheDescriptor? Reminiscent of file descriptor.
That's a good idea. I renamed it
bq. [javadoc issues]
Fixed.
bq. getDirective() should be named getDirectiveWithId() (or getDescriptor() if
you use my suggestion).
OK.
bq. [exception text]
Replace all references to entry etc. with references to
{{PathBasedCacheDescriptor}}.
bq. For implementing equals, how about comparing getClass() rather than
casting? This means it won't take subclasses, which is normally a good thing
(safe symmetric equals).
good idea
bq. extra mortbay.log import
Another auto-add by eclipse. I wonder if I can blacklist this somehow.
separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId
--
Key: HDFS-5213
URL: https://issues.apache.org/jira/browse/HDFS-5213
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-5213-caching.001.patch
Since PathBasedCacheEntry is intended to be a private (implementation) class,
return PathBasedCacheDirectiveWithId from all public APIs instead. Some
other miscellaneous cleanups in the caching RPC stuff.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-5122:
-
Attachment: HDFS-5122.004.patch
rebased to trunk.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
HDFS-5122.003.patch, HDFS-5122.004.patch, HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5066) Inode tree with snapshot information visualization
[ https://issues.apache.org/jira/browse/HDFS-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771015#comment-13771015 ] Jing Zhao commented on HDFS-5066: - bq. What do you mean by make the tool continue the visualization based on the last time result? Here I mean after visualizing the first 500 nodes, we can continue the visualization for the remaining nodes (maybe starting from some specific node that has been visualized). But this is a very hard-to-define functionality and also difficult to implement. So for now maybe we can first move the visualization functionality to SnapshotTestHelper, so that developers can use it for their unit test debugging (just like we use the dumpTree method before)? We can think more about how to expose this feature to outside as the next step. Inode tree with snapshot information visualization --- Key: HDFS-5066 URL: https://issues.apache.org/jira/browse/HDFS-5066 Project: Hadoop HDFS Issue Type: Improvement Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor Attachments: HDFS-5066.v1.patch, HDFS-5066.v2.patch, visnap.png It would be nice to be able to visualize snapshot information, in order to ease the understanding of related data structures. We can generate graph from in memory inode links. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5223) Allow edit log/fsimage format changes without changing layout version
[ https://issues.apache.org/jira/browse/HDFS-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771059#comment-13771059 ] Aaron T. Myers commented on HDFS-5223: -- I was chatting about this informally with [~tlipcon] a day or two ago, and we came up with the following two alternative implementations: # Introduce a new separate NN metadata version number which is decoupled from the existing layout version. We will allow the NN to start up if its NN metadata version number is higher than what's in the fsimage/edit log headers without requiring the '-upgrade' flag. From now on the addition of new edit log opcodes would increment the NN metadata version, and we would require that changes made to the format of existing fsimage/edit log entries be done in a backward compatible fashion. We would freeze the existing layout version number and from now on only increment this in the case of more fundamental NN metadata version changes. # Introduce a set of NN metadata format feature flags which can be enabled or disabled by the admin at runtime. These feature flags could be enabled/disabled entirely independently, so we would move away from a strictly-increasing NN metadata version number. The fsimage and edit log header would be changed to enumerate which of these features were enabled. We will allow the NN to start up only if its software supports the full set of features identified in the fsimage/edit log headers. I'd love to solicit others thoughts/feedback on these proposals, or suggest an alternative if you have one. Allow edit log/fsimage format changes without changing layout version - Key: HDFS-5223 URL: https://issues.apache.org/jira/browse/HDFS-5223 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.1.1-beta Reporter: Aaron T. Myers Currently all HDFS on-disk formats are version by the single layout version. This means that even for changes which might be backward compatible, like the addition of a new edit log op code, we must go through the full `namenode -upgrade' process which requires coordination with DNs, etc. HDFS should support a lighter weight alternative. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4971) Move IO operations out of locking in OpenFileCtx
[
https://issues.apache.org/jira/browse/HDFS-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771070#comment-13771070
]
Hadoop QA commented on HDFS-4971:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603874/HDFS-4971.004.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4997//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4997//console
This message is automatically generated.
Move IO operations out of locking in OpenFileCtx
Key: HDFS-4971
URL: https://issues.apache.org/jira/browse/HDFS-4971
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-4971.000.patch, HDFS-4971.001.patch,
HDFS-4971.002.patch, HDFS-4971.003.patch, HDFS-4971.004.patch
Currently some IO operations (such as writing data to HDFS and dumping to
local disk) in OpenFileCtx may hold a lock which can block processing
incoming writing requests. This jira aims to optimize OpenFileCtx and move
the IO operations out of the locking.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5223) Allow edit log/fsimage format changes without changing layout version
[
https://issues.apache.org/jira/browse/HDFS-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771100#comment-13771100
]
Todd Lipcon commented on HDFS-5223:
---
To expand a little bit on Aaron's summary of our discussion above.
*Proposal 1*:
- note that we already include a version number in the header of the edit log
and image formats. So, within a single image or edits directories, you might
now have different edit log segments or images with different version numbers
-- the ones written post-upgrade would have a higher version number.
- note that this allows for in-place software upgrade, but not in-place
software downgrade. Once you've written an edit log with the new version, you
couldn't downgrade the NN back to the previous version, because it would refuse
to read the higher-versioned edit log segment.
bq. and we would require that changes made to the format of existing
fsimage/edit log entries be done in a backward compatible fashion
This isn't quite the case -- because the new edit log segments would have a new
version number, we have the same ability to evolve opcodes as today. I verified
with Aaron that he mis-stated this above.
*Proposal 2*:
- This is basically the way that file systems such as ext3 handle version
compatibility. Every ext3 filesystem's superblock contains a set of flags which
determine which features have been enabled for it. Similarly, we'd add
something to the edit log and fsimage headers with a set of feature names.
Here's the docs from Documentation/filesystems/ext2.txt in the kernel tree:
{code}
These feature flags have specific meanings for the kernel as follows:
A COMPAT flag indicates that a feature is present in the filesystem,
but the on-disk format is 100% compatible with older on-disk formats, so
a kernel which didn't know anything about this feature could read/write
the filesystem without any chance of corrupting the filesystem (or even
making it inconsistent). This is essentially just a flag which says
this filesystem has a (hidden) feature that the kernel or e2fsck may
want to be aware of (more on e2fsck and feature flags later). The ext3
HAS_JOURNAL feature is a COMPAT flag because the ext3 journal is simply
a regular file with data blocks in it so the kernel does not need to
take any special notice of it if it doesn't understand ext3 journaling.
An RO_COMPAT flag indicates that the on-disk format is 100% compatible
with older on-disk formats for reading (i.e. the feature does not change
the visible on-disk format). However, an old kernel writing to such a
filesystem would/could corrupt the filesystem, so this is prevented. The
most common such feature, SPARSE_SUPER, is an RO_COMPAT feature because
sparse groups allow file data blocks where superblock/group descriptor
backups used to live, and ext2_free_blocks() refuses to free these blocks,
which would leading to inconsistent bitmaps. An old kernel would also
get an error if it tried to free a series of blocks which crossed a group
boundary, but this is a legitimate layout in a SPARSE_SUPER filesystem.
An INCOMPAT flag indicates the on-disk format has changed in some
way that makes it unreadable by older kernels, or would otherwise
cause a problem if an old kernel tried to mount it. FILETYPE is an
INCOMPAT flag because older kernels would think a filename was longer
than 256 characters, which would lead to corrupt directory listings.
The COMPRESSION flag is an obvious INCOMPAT flag - if the kernel
doesn't understand compression, you would just get garbage back from
read() instead of it automatically decompressing your data. The ext3
RECOVER flag is needed to prevent a kernel which does not understand the
ext3 journal from mounting the filesystem without replaying the journal.
{code}
This would allow us to do rolling upgrades, run mixed-version clusters, and
still retain the ability to roll back to a prior version until the new feature
was used. So, to take the example of a feature like snapshots which required a
metadata change, the admin workflow would be:
# Shutdown standby node
# Upgrade standby software version
# Start standby node, failover to it
# Shutdown and upgrade the old active, start it back up.
# Note: at this point, the format for the edit logs and images is identical to
the pre-upgrade format, so the user could still roll back. Trying to create a
snapshot at this point would fail with an error like Snapshots not enabled for
this filesystem. Run dfsadmin -enableFeature snapshots to enable
# User runs the above command, which forces an edit log roll. The new edit logs
contain the flag indicating that snapshots are enabled, and may use the new
opcodes (or add new fields to the old opcodes as necessary)
If the explicit enable doesn't sit well with people, we could also add a
slightly simpler version like -enableAllNewFeatures or whatever, which a user
can use after an upgrade
[jira] [Commented] (HDFS-5213) separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId
[ https://issues.apache.org/jira/browse/HDFS-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771137#comment-13771137 ] Colin Patrick McCabe commented on HDFS-5213: bq. I still want to see Paths, not Strings, for the DFS methods involving cache directives. It's non-standard and weird for the reasons I mentioned before, and DFS has error checking for being passed a URI for a different FileSystem. After considering this, I agree. we can have the paths normalized and absolutized in DistributedFileSystem. I started doing this, but it got to be too many changes for one patch. It's a little tricky how it interacts with directive validation. Let's do this in a future frontend patch. separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId -- Key: HDFS-5213 URL: https://issues.apache.org/jira/browse/HDFS-5213 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-5213-caching.001.patch Since PathBasedCacheEntry is intended to be a private (implementation) class, return PathBasedCacheDirectiveWithId from all public APIs instead. Some other miscellaneous cleanups in the caching RPC stuff. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5213) separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId
[ https://issues.apache.org/jira/browse/HDFS-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5213: --- Attachment: HDFS-5213-caching.003.patch separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId -- Key: HDFS-5213 URL: https://issues.apache.org/jira/browse/HDFS-5213 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-5213-caching.001.patch, HDFS-5213-caching.003.patch Since PathBasedCacheEntry is intended to be a private (implementation) class, return PathBasedCacheDirectiveWithId from all public APIs instead. Some other miscellaneous cleanups in the caching RPC stuff. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5031) BlockScanner scans the block multiple times and on restart scans everything
[ https://issues.apache.org/jira/browse/HDFS-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771157#comment-13771157 ] Arpit Agarwal commented on HDFS-5031: - Okay that makes sense. The patch looks good, I will commit this shortly. Thanks! BlockScanner scans the block multiple times and on restart scans everything --- Key: HDFS-5031 URL: https://issues.apache.org/jira/browse/HDFS-5031 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0, 2.1.0-beta Reporter: Vinay Assignee: Vinay Attachments: HDFS-5031.patch, HDFS-5031.patch, HDFS-5031.patch BlockScanner scans the block twice, also on restart of datanode scans everything. Steps: 1. Write blocks with interval of more than 5 seconds. write new block on completion of scan for written block. Each time datanode scans new block, it also scans, previous block which is already scanned. Now after restart, datanode scans all blocks again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5031) BlockScanner scans the block multiple times and on restart scans everything
[ https://issues.apache.org/jira/browse/HDFS-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771199#comment-13771199 ] Arpit Agarwal commented on HDFS-5031: - I have committed this to trunk and branch-2. Thanks for the submitting the patch and your patience, Vinay. BlockScanner scans the block multiple times and on restart scans everything --- Key: HDFS-5031 URL: https://issues.apache.org/jira/browse/HDFS-5031 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0, 2.1.0-beta Reporter: Vinay Assignee: Vinay Attachments: HDFS-5031.patch, HDFS-5031.patch, HDFS-5031.patch BlockScanner scans the block twice, also on restart of datanode scans everything. Steps: 1. Write blocks with interval of more than 5 seconds. write new block on completion of scan for written block. Each time datanode scans new block, it also scans, previous block which is already scanned. Now after restart, datanode scans all blocks again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5031) BlockScanner scans the block multiple times and on restart scans everything
[ https://issues.apache.org/jira/browse/HDFS-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771186#comment-13771186 ] Hudson commented on HDFS-5031: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4436 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4436/]) HDFS-5031. BlockScanner scans the block multiple times. (Vinay via Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1524553) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RollingLogs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RollingLogsImpl.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeBlockScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java BlockScanner scans the block multiple times and on restart scans everything --- Key: HDFS-5031 URL: https://issues.apache.org/jira/browse/HDFS-5031 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0, 2.1.0-beta Reporter: Vinay Assignee: Vinay Attachments: HDFS-5031.patch, HDFS-5031.patch, HDFS-5031.patch BlockScanner scans the block twice, also on restart of datanode scans everything. Steps: 1. Write blocks with interval of more than 5 seconds. write new block on completion of scan for written block. Each time datanode scans new block, it also scans, previous block which is already scanned. Now after restart, datanode scans all blocks again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771204#comment-13771204
]
Jing Zhao commented on HDFS-5122:
-
+1 for the new patch. I will commit it soon.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
HDFS-5122.003.patch, HDFS-5122.004.patch, HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4320) Add a separate configuration for namenode rpc address instead of only using fs.default.name
[ https://issues.apache.org/jira/browse/HDFS-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu updated HDFS-4320: Affects Version/s: 2.1.1-beta 3.0.0 Add a separate configuration for namenode rpc address instead of only using fs.default.name --- Key: HDFS-4320 URL: https://issues.apache.org/jira/browse/HDFS-4320 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, namenode Affects Versions: 1.1.0, 3.0.0, 2.1.1-beta Reporter: Mostafa Elhemali Assignee: Mostafa Elhemali Fix For: 1.2.0 Attachments: HDFS-4320.branch-1.2.patch, HDFS-4320.branch-1.3.patch, HDFS-4320.patch, HDFS-4320.trunk.2.patch, HDFS-4320-trunk.3.patch, HDFS-4320-trunk.4.patch, HDFS-4320.trunk.patch When NameNode starts up, it tries to find out its address by looking at the fs.default.name configuration key instead of using its own keys. This breaks scenarios where we try to configure a Hadoop cluster that uses a different default file system than DFS, but still try to prop up namenode and datanode services as a secondary file system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771174#comment-13771174
]
Hadoop QA commented on HDFS-5122:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603868/HDFS-5122.004.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4996//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4996//console
This message is automatically generated.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
HDFS-5122.003.patch, HDFS-5122.004.patch, HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5122) Support failover and retry in WebHdfsFileSystem for NN HA
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-5122:
Summary: Support failover and retry in WebHdfsFileSystem for NN HA (was:
WebHDFS should support logical service names in URIs)
Support failover and retry in WebHdfsFileSystem for NN HA
-
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
HDFS-5122.003.patch, HDFS-5122.004.patch, HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5213) separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId
[ https://issues.apache.org/jira/browse/HDFS-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771209#comment-13771209 ] Andrew Wang commented on HDFS-5213: --- +1 will commit shortly, thanks Colin! I'll file a follow-on JIRA for the String-to-Path change. separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId -- Key: HDFS-5213 URL: https://issues.apache.org/jira/browse/HDFS-5213 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-5213-caching.001.patch, HDFS-5213-caching.003.patch Since PathBasedCacheEntry is intended to be a private (implementation) class, return PathBasedCacheDirectiveWithId from all public APIs instead. Some other miscellaneous cleanups in the caching RPC stuff. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-5181) Fail-over support for HA cluster in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-5181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-5181. - Resolution: Duplicate Fail-over support for HA cluster in WebHDFS Key: HDFS-5181 URL: https://issues.apache.org/jira/browse/HDFS-5181 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Reporter: Haohui Mai Assignee: Haohui Mai HDFS-5122 only teaches WebHDFS client to recognize the logical name in HA clusters. The WebHDFS client should implement fail-over mechanisms in order to fully support HA clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5122) Support failover and retry in WebHdfsFileSystem for NN HA
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-5122:
Description:
Bug reported by [~arpitgupta]:
If the dfs.nameservices is set to arpit,
{code}
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work. You have to provide the exact active namenode hostname. On an HA
cluster using dfs client one should not need to provide the active nn hostname.
To fix this, we try to
1) let WebHdfsFileSystem support logical NN service name
2) add failover_and_retry functionality in WebHdfsFileSystem for NN HA
was:
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster using
dfs client one should not need to provide the active nn hostname
Support failover and retry in WebHdfsFileSystem for NN HA
-
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
HDFS-5122.003.patch, HDFS-5122.004.patch, HDFS-5122.patch
Bug reported by [~arpitgupta]:
If the dfs.nameservices is set to arpit,
{code}
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work. You have to provide the exact active namenode hostname. On an
HA cluster using dfs client one should not need to provide the active nn
hostname.
To fix this, we try to
1) let WebHdfsFileSystem support logical NN service name
2) add failover_and_retry functionality in WebHdfsFileSystem for NN HA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5031) BlockScanner scans the block multiple times and on restart scans everything
[ https://issues.apache.org/jira/browse/HDFS-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5031: Resolution: Fixed Fix Version/s: 2.3.0 Status: Resolved (was: Patch Available) BlockScanner scans the block multiple times and on restart scans everything --- Key: HDFS-5031 URL: https://issues.apache.org/jira/browse/HDFS-5031 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0, 2.1.0-beta Reporter: Vinay Assignee: Vinay Fix For: 2.3.0 Attachments: HDFS-5031.patch, HDFS-5031.patch, HDFS-5031.patch BlockScanner scans the block twice, also on restart of datanode scans everything. Steps: 1. Write blocks with interval of more than 5 seconds. write new block on completion of scan for written block. Each time datanode scans new block, it also scans, previous block which is already scanned. Now after restart, datanode scans all blocks again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5224) Refactor PathBasedCache* methods to use a Path rather than a String
Andrew Wang created HDFS-5224: - Summary: Refactor PathBasedCache* methods to use a Path rather than a String Key: HDFS-5224 URL: https://issues.apache.org/jira/browse/HDFS-5224 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS-4949 Reporter: Andrew Wang Assignee: Colin Patrick McCabe As discussed in HDFS-5213, we should refactor PathBasedCacheDirective and related methods in DistributedFileSystem to use a Path to represent paths to cache, rather than a String. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-5213) separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId
[ https://issues.apache.org/jira/browse/HDFS-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-5213. --- Resolution: Fixed Fix Version/s: HDFS-4949 Committed, thanks again Colin. Follow-on JIRA for string-to-path filed at HDFS-5224. separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId -- Key: HDFS-5213 URL: https://issues.apache.org/jira/browse/HDFS-5213 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: HDFS-4949 Attachments: HDFS-5213-caching.001.patch, HDFS-5213-caching.003.patch Since PathBasedCacheEntry is intended to be a private (implementation) class, return PathBasedCacheDirectiveWithId from all public APIs instead. Some other miscellaneous cleanups in the caching RPC stuff. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) Support failover and retry in WebHdfsFileSystem for NN HA
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771230#comment-13771230
]
Hudson commented on HDFS-5122:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #4437 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4437/])
HDFS-5122. Support failover and retry in WebHdfsFileSystem for NN HA.
Contributed by Haohui Mai. (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1524562)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSForHA.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/WebHdfsTestUtil.java
Support failover and retry in WebHdfsFileSystem for NN HA
-
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
HDFS-5122.003.patch, HDFS-5122.004.patch, HDFS-5122.patch
Bug reported by [~arpitgupta]:
If the dfs.nameservices is set to arpit,
{code}
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work. You have to provide the exact active namenode hostname. On an
HA cluster using dfs client one should not need to provide the active nn
hostname.
To fix this, we try to
1) let WebHdfsFileSystem support logical NN service name
2) add failover_and_retry functionality in WebHdfsFileSystem for NN HA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5122) Support failover and retry in WebHdfsFileSystem for NN HA
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-5122:
Resolution: Fixed
Fix Version/s: 2.3.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the work, [~wheat9]! I've committed this to trunk and branch-2.
Support failover and retry in WebHdfsFileSystem for NN HA
-
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Fix For: 2.3.0
Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
HDFS-5122.003.patch, HDFS-5122.004.patch, HDFS-5122.patch
Bug reported by [~arpitgupta]:
If the dfs.nameservices is set to arpit,
{code}
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work. You have to provide the exact active namenode hostname. On an
HA cluster using dfs client one should not need to provide the active nn
hostname.
To fix this, we try to
1) let WebHdfsFileSystem support logical NN service name
2) add failover_and_retry functionality in WebHdfsFileSystem for NN HA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) Support failover and retry in WebHdfsFileSystem for NN HA
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771275#comment-13771275
]
Hudson commented on HDFS-5122:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #4438 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4438/])
Move HDFS-5122 from Release 2.1.1-beta to Release 2.3.0 in CHANGES.txt (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1524581)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Support failover and retry in WebHdfsFileSystem for NN HA
-
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Fix For: 2.3.0
Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
HDFS-5122.003.patch, HDFS-5122.004.patch, HDFS-5122.patch
Bug reported by [~arpitgupta]:
If the dfs.nameservices is set to arpit,
{code}
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work. You have to provide the exact active namenode hostname. On an
HA cluster using dfs client one should not need to provide the active nn
hostname.
To fix this, we try to
1) let WebHdfsFileSystem support logical NN service name
2) add failover_and_retry functionality in WebHdfsFileSystem for NN HA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-5221) hftp: does not work with HA NN configuration
[ https://issues.apache.org/jira/browse/HDFS-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-5221. --- Resolution: Duplicate Duping to HDFS-5123. Joep, it seems likely that one of your two approaches will be implemented (probably the first). hftp: does not work with HA NN configuration Key: HDFS-5221 URL: https://issues.apache.org/jira/browse/HDFS-5221 Project: Hadoop HDFS Issue Type: Bug Components: ha, hdfs-client Affects Versions: 2.0.5-alpha Reporter: Joep Rottinghuis Priority: Blocker When copying data between clusters of significant different version (say from Hadoop 1.x equivalent to Hadoop 2.x) we have to use hftp. When HA is configured, you have to point to a single (active) NN. Now, when the active NN becomes standby, the the hftp: addresses will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4990) Block placement for storage types
[ https://issues.apache.org/jira/browse/HDFS-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771314#comment-13771314 ] Arpit Agarwal commented on HDFS-4990: - Update looks good, thanks! Block placement for storage types - Key: HDFS-4990 URL: https://issues.apache.org/jira/browse/HDFS-4990 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Suresh Srinivas Assignee: Tsz Wo (Nicholas), SZE Fix For: Heterogeneous Storage (HDFS-2832) Attachments: h4990_20130909.patch, h4990_20130916.patch, h4990_20130917b.patch, h4990_20130917c.patch, h4990_20130917.patch, h4990_20130918b.patch, h4990_20130918.patch Currently block location for writes are made based on: # Datanode load (number of transceivers) # Space left on datanode # Topology With storage abstraction, namenode must choose a storage instead of a datanode for block placement. It also needs to consider storage type, load on the storage etc. As an additional benefit, currently HDFS support heterogeneous nodes (nodes with different number of spindles etc.) poorly. This work should help solve that issue as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over agin
Roman Shaposhnik created HDFS-5225:
--
Summary: datanode keeps logging the same 'is no longer in the
dataset' message over and over agin
Key: HDFS-5225
URL: https://issues.apache.org/jira/browse/HDFS-5225
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.1.1-beta
Reporter: Roman Shaposhnik
I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
configuration: 4 nodes fully distributed cluster with security on.
All of a sudden my DN ate up all of the space in /var/log logging the following
message repeatedly:
{noformat}
2013-09-18 20:51:12,046 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer in
the dataset
{noformat}
It wouldn't answer to a jstack and jstack -F ended up being useless.
Here's what I was able to find in the NameNode logs regarding this block ID:
{noformat}
fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
BP-1884637155-10.224.158.152-1379524544853
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
updated: 10.224.158.152:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
updated: 10.34.74.206:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
updated: 10.83.107.80:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,899 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:17,904 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
successfully to BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
2013-09-18 18:03:18,540 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:18,548 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370)
successfully to BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1371
2013-09-18 18:03:26,150 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073742189_1371 10.83.107.80:1004 10.34.74.206:1004 10.224.158.152:1004
2013-09-18 18:03:26,847 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.34.74.206:1004 to delete [blk_1073742178_1359,
blk_1073742183_1362, blk_1073742184_1363, blk_1073742186_1366,
blk_1073742188_1368, blk_1073742189_1371]
2013-09-18 18:03:29,848 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.224.158.152:1004 to delete [blk_1073742177_1353,
blk_1073742178_1359, blk_1073742179_1355, blk_1073742180_1356,
blk_1073742181_1358, blk_1073742182_1361, blk_1073742185_1364,
blk_1073742187_1367, blk_1073742188_1368, blk_1073742189_1371]
2013-09-18 18:03:29,848 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.83.107.80:1004 to delete [blk_1073742177_1353,
blk_1073742178_1359, blk_1073742179_1355, blk_1073742180_1356,
blk_1073742181_1358, blk_1073742182_1361, blk_1073742183_1362,
blk_1073742184_1363, blk_1073742185_1364, blk_1073742186_1366,
blk_1073742187_1367, blk_1073742188_1368, blk_1073742189_1371]
[jira] [Commented] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over agin
[
https://issues.apache.org/jira/browse/HDFS-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771334#comment-13771334
]
Colin Patrick McCabe commented on HDFS-5225:
I haven't had time to look into this fully, but I suspect that it's some kind
of race condition in the BlockPoolSliceScanner. Whatever it is, it certainly
did lead to a lot of logs getting generated (there were gigabytes of that is
no longer in the dataset message). The datanode stayed in this state until
restarted.
datanode keeps logging the same 'is no longer in the dataset' message over
and over agin
Key: HDFS-5225
URL: https://issues.apache.org/jira/browse/HDFS-5225
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.1.1-beta
Reporter: Roman Shaposhnik
I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
configuration: 4 nodes fully distributed cluster with security on.
All of a sudden my DN ate up all of the space in /var/log logging the
following message repeatedly:
{noformat}
2013-09-18 20:51:12,046 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer
in the dataset
{noformat}
It wouldn't answer to a jstack and jstack -F ended up being useless.
Here's what I was able to find in the NameNode logs regarding this block ID:
{noformat}
fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
BP-1884637155-10.224.158.152-1379524544853
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.224.158.152:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.34.74.206:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.83.107.80:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,899 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:17,904 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
2013-09-18 18:03:18,540 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:18,548 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1371
2013-09-18 18:03:26,150 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073742189_1371 10.83.107.80:1004 10.34.74.206:1004 10.224.158.152:1004
2013-09-18 18:03:26,847 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.34.74.206:1004 to delete [blk_1073742178_1359,
blk_1073742183_1362, blk_1073742184_1363, blk_1073742186_1366,
blk_1073742188_1368, blk_1073742189_1371]
2013-09-18 18:03:29,848 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
[jira] [Created] (HDFS-5226) Trash::moveToTrash doesn't work across multiple namespace
Joep Rottinghuis created HDFS-5226: -- Summary: Trash::moveToTrash doesn't work across multiple namespace Key: HDFS-5226 URL: https://issues.apache.org/jira/browse/HDFS-5226 Project: Hadoop HDFS Issue Type: Bug Components: federation Affects Versions: 2.0.5-alpha Reporter: Joep Rottinghuis Priority: Blocker Trash has introduced new static method moveToAppropriateTrash which resolves to right filesystem. To be API compatible we need to check if Trash::moveToTrash can do what moveToAppropriateTrash does so that downstream users need not change code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over again
[
https://issues.apache.org/jira/browse/HDFS-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated HDFS-5225:
---
Summary: datanode keeps logging the same 'is no longer in the dataset'
message over and over again (was: datanode keeps logging the same 'is no
longer in the dataset' message over and over agin)
datanode keeps logging the same 'is no longer in the dataset' message over
and over again
-
Key: HDFS-5225
URL: https://issues.apache.org/jira/browse/HDFS-5225
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.1.1-beta
Reporter: Roman Shaposhnik
I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
configuration: 4 nodes fully distributed cluster with security on.
All of a sudden my DN ate up all of the space in /var/log logging the
following message repeatedly:
{noformat}
2013-09-18 20:51:12,046 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer
in the dataset
{noformat}
It wouldn't answer to a jstack and jstack -F ended up being useless.
Here's what I was able to find in the NameNode logs regarding this block ID:
{noformat}
fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
BP-1884637155-10.224.158.152-1379524544853
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.224.158.152:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.34.74.206:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.83.107.80:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,899 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:17,904 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
2013-09-18 18:03:18,540 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:18,548 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1371
2013-09-18 18:03:26,150 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073742189_1371 10.83.107.80:1004 10.34.74.206:1004 10.224.158.152:1004
2013-09-18 18:03:26,847 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.34.74.206:1004 to delete [blk_1073742178_1359,
blk_1073742183_1362, blk_1073742184_1363, blk_1073742186_1366,
blk_1073742188_1368, blk_1073742189_1371]
2013-09-18 18:03:29,848 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.224.158.152:1004 to delete [blk_1073742177_1353,
blk_1073742178_1359, blk_1073742179_1355, blk_1073742180_1356,
blk_1073742181_1358, blk_1073742182_1361,
[jira] [Commented] (HDFS-5226) Trash::moveToTrash doesn't work across multiple namespace
[ https://issues.apache.org/jira/browse/HDFS-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771341#comment-13771341 ] Joep Rottinghuis commented on HDFS-5226: This manifested itself by Pig being unable to move directories to Trash. Trash::moveToTrash doesn't work across multiple namespace - Key: HDFS-5226 URL: https://issues.apache.org/jira/browse/HDFS-5226 Project: Hadoop HDFS Issue Type: Bug Components: federation Affects Versions: 2.0.5-alpha Reporter: Joep Rottinghuis Priority: Blocker Trash has introduced new static method moveToAppropriateTrash which resolves to right filesystem. To be API compatible we need to check if Trash::moveToTrash can do what moveToAppropriateTrash does so that downstream users need not change code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5227) Namenode.getAddress() should return namenode rpc-address
Chuan Liu created HDFS-5227:
---
Summary: Namenode.getAddress() should return namenode rpc-address
Key: HDFS-5227
URL: https://issues.apache.org/jira/browse/HDFS-5227
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Currently, {{Namenode.getAddress(Configuration conf)}} will return default file
system address as its result. The correct behavior should be returning config
value of dfs.namenode.rpc-address if it presents in the configurations.
Otherwise namenode will fail to start if the default file system is configured
to another file system other than the one running in the cluster. We have a
similar issue in 1.0 code base. The JIRA is HDFS-4320. The previous JIRA is
closed and we cannot open it. Thus create a new one to track the issue in trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5227) Namenode.getAddress() should return namenode rpc-address
[
https://issues.apache.org/jira/browse/HDFS-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chuan Liu updated HDFS-5227:
Status: Patch Available (was: Open)
Namenode.getAddress() should return namenode rpc-address
Key: HDFS-5227
URL: https://issues.apache.org/jira/browse/HDFS-5227
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HDFS-5227-trunk.patch
Currently, {{Namenode.getAddress(Configuration conf)}} will return default
file system address as its result. The correct behavior should be returning
config value of dfs.namenode.rpc-address if it presents in the
configurations. Otherwise namenode will fail to start if the default file
system is configured to another file system other than the one running in the
cluster. We have a similar issue in 1.0 code base. The JIRA is HDFS-4320. The
previous JIRA is closed and we cannot open it. Thus create a new one to track
the issue in trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5227) Namenode.getAddress() should return namenode rpc-address
[
https://issues.apache.org/jira/browse/HDFS-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chuan Liu updated HDFS-5227:
Attachment: HDFS-5227-trunk.patch
Attach a patch.
Namenode.getAddress() should return namenode rpc-address
Key: HDFS-5227
URL: https://issues.apache.org/jira/browse/HDFS-5227
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HDFS-5227-trunk.patch
Currently, {{Namenode.getAddress(Configuration conf)}} will return default
file system address as its result. The correct behavior should be returning
config value of dfs.namenode.rpc-address if it presents in the
configurations. Otherwise namenode will fail to start if the default file
system is configured to another file system other than the one running in the
cluster. We have a similar issue in 1.0 code base. The JIRA is HDFS-4320. The
previous JIRA is closed and we cannot open it. Thus create a new one to track
the issue in trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5167) Add metrics about the NameNode retry cache
[ https://issues.apache.org/jira/browse/HDFS-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated HDFS-5167: - Attachment: HDFS-5167.4.patch Updated to pass tests. Add metrics about the NameNode retry cache -- Key: HDFS-5167 URL: https://issues.apache.org/jira/browse/HDFS-5167 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, namenode Affects Versions: 3.0.0 Reporter: Jing Zhao Priority: Minor Attachments: HDFS-5167.1.patch, HDFS-5167.2.patch, HDFS-5167.3.patch, HDFS-5167.4.patch It will be helpful to have metrics in NameNode about the retry cache, such as the retry count etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4320) Add a separate configuration for namenode rpc address instead of only using fs.default.name
[ https://issues.apache.org/jira/browse/HDFS-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771368#comment-13771368 ] Chuan Liu commented on HDFS-4320: - I created a new JIRA HDFS-5227 to track the issue in trunk. Add a separate configuration for namenode rpc address instead of only using fs.default.name --- Key: HDFS-4320 URL: https://issues.apache.org/jira/browse/HDFS-4320 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, namenode Affects Versions: 1.1.0, 3.0.0, 2.1.1-beta Reporter: Mostafa Elhemali Assignee: Mostafa Elhemali Fix For: 1.2.0 Attachments: HDFS-4320.branch-1.2.patch, HDFS-4320.branch-1.3.patch, HDFS-4320.patch, HDFS-4320.trunk.2.patch, HDFS-4320-trunk.3.patch, HDFS-4320-trunk.4.patch, HDFS-4320.trunk.patch When NameNode starts up, it tries to find out its address by looking at the fs.default.name configuration key instead of using its own keys. This breaks scenarios where we try to configure a Hadoop cluster that uses a different default file system than DFS, but still try to prop up namenode and datanode services as a secondary file system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5191) revisit zero-copy API in FSDataInputStream to make it more intuitive
[
https://issues.apache.org/jira/browse/HDFS-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771369#comment-13771369
]
Chris Nauroth commented on HDFS-5191:
-
I reviewed the current API and caught up on all prior discussion here and in
HDFS-4953. [~cmccabe], thank you for incorporating the feedback. I'm going to
focus on the interface here. I'll review the implementation details soon too.
I like to review APIs by writing a little code against them. This is what I
came up with (imports trimmed for brevity):
{code}
class Main extends Configured implements Tool {
private static final int BUFFER_MAX_LENGTH = 4 * 1024 * 1024; // 4 MB
private static final Log LOG = LogFactory.getLog(Main.class);
@Override
public int run(String[] args) {
FileSystem fs = null;
FSDataInputStream is = null;
try {
fs = FileSystem.get(this.getConf());
is = fs.open(new Path(args[0]));
ByteBufferFactory bbf = new SimpleByteBufferFactory();
EnumSetReadOption readOpts = EnumSet.noneOf(ReadOption.class);
for (;;) {
ByteBuffer bb = is.read(bbf, BUFFER_MAX_LENGTH, readOpts);
if (bb == null) break; // EOF
// Use data from the buffer here.
bbf.putBuffer(bb);
}
} catch (IOException e) {
LOG.error(Unexpected IOException, e);
} finally {
IOUtils.cleanup(LOG, is, fs);
}
return 0;
}
public static void main(String[] args) throws Exception {
System.exit(ToolRunner.run(new Main(), args));
}
}
{code}
Can someone check that this code (written by someone looking at the API for the
first time) is correct? If so, then that's a good sign that we're on the right
track. :-) I think this code sample shows that most of the earlier feedback
has been addressed. Specifically:
# It's a single interface for client read code for both the cached and uncached
case. I didn't need to check flags or catch a special exception or downcast to
an HDFS class to handle copying vs. zero-copy.
# There is generally less code for the client to write, because there are fewer
things that the client needs to control. (i.e. no {{setAllowShortReads}})
# I did not need to preallocate a fallback buffer that wouldn't be used in the
mmap'd case.
# I did not need to catch exceptions for main flow control.
# The {{ByteBufferFactory}} interface would allow the client to control
ownership of direct buffers for fallback (and the {{SimpleByteBufferFactory}}
ships a default implementation that does this).
# There is no sharing of mutable state between implicitly connected objects
(streams and cursors).
# It looks close to the kind of code sample [~owen.omalley] wanted to achieve
in the comments of HDFS-4953.
# There had been discussion of returning array of {{ByteBuffer}} vs. returning
a single {{ByteBuffer}} with possibility of short read. My preference is for
the latter. The former would have required me to write another nested loop. I
need to code for short read anyway in case the final read before EOF doesn't
match my desired read length.
I think the last remaining open question is around the need for a client to
check if zero-copy is available and potentially run a different code path. One
way we could achieve that now is by adding a {{RejectingByteBufferFactory}}
that always throws, and clients that want that behavior can use it instead of
{{SimpleByteBufferFactory}}. Does anyone have a concrete use case for this?
Without a concrete use case, it's hard to say if the interface is sufficient.
Here are some suggestions, mostly minor adjustments on top of the current patch:
# Shall we add overloads of {{FSDataInputStream#read}} that don't require the
caller to pass {{ByteBufferFactory}} (assumes caller wants a new
{{SimpleByteBufferFactory}}) and don't require the caller to pass read opts
(assumes none)?
# Can we rename {{ByteBufferFactory}} to {{ByteBufferPool}}? [~vicaya] had
made a similar suggestion. Factory implies per-call instance creation and
pool communicates that it needs to control the lifecycle of instances.
# Can we move those classes to the .io sub-package? They aren't coupled to
anything in .fs.
# We need to fully document the expected contract of {{ByteBufferPool}} for
implementers. Some factors to consider:
## Thread-safety - Is it required for the implementation to guarantee
thread-safety internally? (I assume thread-safety is only required if the
caller intends to use the same one from multiple threads. Whatever the answer,
let's document it.)
## Null - Is null a legal return value? Is it possible for callers to pass
null arguments? (Ideally, null is forbidden, but whatever the answer, let's
document it.)
## Bounds-checking - What should implementers do if given a negative length?
(Runtime exception?)
# When the {{FSDataInputStream}} is not {{HasEnhancedByteBufferAccess}}, we try
to
[jira] [Commented] (HDFS-5191) revisit zero-copy API in FSDataInputStream to make it more intuitive
[
https://issues.apache.org/jira/browse/HDFS-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771412#comment-13771412
]
Colin Patrick McCabe commented on HDFS-5191:
My feeling is that we should support passing a null {{ByteBufferFactory}} to
mean don't create fallback {{ByteBuffers}}. Using a
{{RejectingByteBufferFactory}} is another reasonable choice, but it would
require more typing for some people.
I'll add an overload for the empty EnumSet case.
ByteBufferPool is a better name, I agree.
I suppose, given that {{FSDataInputStream}} is in {{org.apache.hadoop.io}},
ByteBufferPool/Factory should be as well.
{{ByteBufferPool}} implementations don't need thread-safety unless multiple
read calls are going to be made in parallel using the same pool. I'll add that
information to the JavaDoc.
I agree that the fallback fallback path is something that still needs to be
done. The problem is, there isn't a very efficient way to do it, since we'd
have to read into a byte array, and then copy to the direct byte buffer. We
could do better, if we could ask the ByteBufferPool for a non-direct buffer.
(i.e., an array-backed buffer). Will this fallback fallback case be common
enough to motivate this kind of API?
The disadvantage of this is that then our read function would sometimes return
direct byte buffers, and sometimes not, which could lead to code working on
local filesystems, and then failing on HDFS (if it tried to call
ByteBuffer#array).
revisit zero-copy API in FSDataInputStream to make it more intuitive
Key: HDFS-5191
URL: https://issues.apache.org/jira/browse/HDFS-5191
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client, libhdfs
Affects Versions: HDFS-4949
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-5191-caching.001.patch
As per the discussion on HDFS-4953, we should revisit the zero-copy API to
make it more intuitive for new users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over again
[
https://issues.apache.org/jira/browse/HDFS-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771417#comment-13771417
]
Tsuyoshi OZAWA commented on HDFS-5225:
--
Which should we dealing with this problem by, log4j setting or source code
level fix? We can add a configuration to limit msg of is no longer in the
dataset if we fix it at source code level.
datanode keeps logging the same 'is no longer in the dataset' message over
and over again
-
Key: HDFS-5225
URL: https://issues.apache.org/jira/browse/HDFS-5225
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.1.1-beta
Reporter: Roman Shaposhnik
I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
configuration: 4 nodes fully distributed cluster with security on.
All of a sudden my DN ate up all of the space in /var/log logging the
following message repeatedly:
{noformat}
2013-09-18 20:51:12,046 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer
in the dataset
{noformat}
It wouldn't answer to a jstack and jstack -F ended up being useless.
Here's what I was able to find in the NameNode logs regarding this block ID:
{noformat}
fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
BP-1884637155-10.224.158.152-1379524544853
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.224.158.152:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.34.74.206:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.83.107.80:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,899 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:17,904 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
2013-09-18 18:03:18,540 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:18,548 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1371
2013-09-18 18:03:26,150 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073742189_1371 10.83.107.80:1004 10.34.74.206:1004 10.224.158.152:1004
2013-09-18 18:03:26,847 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.34.74.206:1004 to delete [blk_1073742178_1359,
blk_1073742183_1362, blk_1073742184_1363, blk_1073742186_1366,
blk_1073742188_1368, blk_1073742189_1371]
2013-09-18 18:03:29,848 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.224.158.152:1004 to delete [blk_1073742177_1353,
blk_1073742178_1359, blk_1073742179_1355, blk_1073742180_1356,
[jira] [Commented] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over again
[
https://issues.apache.org/jira/browse/HDFS-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771422#comment-13771422
]
Roman Shaposhnik commented on HDFS-5225:
The problem here is really not log overflow. That can be dealt with (as I
mentioned). The problem here is that the observed behavior seems to indicated a
deeper (potentially significant) problem along the lines of what [~cmccabe] has
mentioned.
datanode keeps logging the same 'is no longer in the dataset' message over
and over again
-
Key: HDFS-5225
URL: https://issues.apache.org/jira/browse/HDFS-5225
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.1.1-beta
Reporter: Roman Shaposhnik
I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
configuration: 4 nodes fully distributed cluster with security on.
All of a sudden my DN ate up all of the space in /var/log logging the
following message repeatedly:
{noformat}
2013-09-18 20:51:12,046 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer
in the dataset
{noformat}
It wouldn't answer to a jstack and jstack -F ended up being useless.
Here's what I was able to find in the NameNode logs regarding this block ID:
{noformat}
fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
BP-1884637155-10.224.158.152-1379524544853
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.224.158.152:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.34.74.206:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.83.107.80:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,899 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:17,904 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
2013-09-18 18:03:18,540 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:18,548 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1371
2013-09-18 18:03:26,150 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073742189_1371 10.83.107.80:1004 10.34.74.206:1004 10.224.158.152:1004
2013-09-18 18:03:26,847 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.34.74.206:1004 to delete [blk_1073742178_1359,
blk_1073742183_1362, blk_1073742184_1363, blk_1073742186_1366,
blk_1073742188_1368, blk_1073742189_1371]
2013-09-18 18:03:29,848 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.224.158.152:1004 to delete [blk_1073742177_1353,
[jira] [Assigned] (HDFS-5178) DataTransferProtocol changes for supporting mutiple storages per DN
[ https://issues.apache.org/jira/browse/HDFS-5178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du reassigned HDFS-5178: Assignee: Junping Du DataTransferProtocol changes for supporting mutiple storages per DN --- Key: HDFS-5178 URL: https://issues.apache.org/jira/browse/HDFS-5178 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: Heterogeneous Storage (HDFS-2832) Reporter: Junping Du Assignee: Junping Du Per discussion in HDFS-5157, DataTransferProtocol should be updated to add StorageID info in some methods, i.e. writeBlock(), replaceBlock(), etc. After that, BlockReceiver (sender also) and receiveBlock can operate on target storage with new parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5178) DataTransferProtocol changes for supporting mutiple storages per DN
[ https://issues.apache.org/jira/browse/HDFS-5178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771440#comment-13771440 ] Arpit Agarwal commented on HDFS-5178: - Junping, can you please hold off on this patch just a bit? I will be making a few protocol changes as part of HDFS-4988. Thanks. DataTransferProtocol changes for supporting mutiple storages per DN --- Key: HDFS-5178 URL: https://issues.apache.org/jira/browse/HDFS-5178 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: Heterogeneous Storage (HDFS-2832) Reporter: Junping Du Assignee: Junping Du Per discussion in HDFS-5157, DataTransferProtocol should be updated to add StorageID info in some methods, i.e. writeBlock(), replaceBlock(), etc. After that, BlockReceiver (sender also) and receiveBlock can operate on target storage with new parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5178) DataTransferProtocol changes for supporting mutiple storages per DN
[ https://issues.apache.org/jira/browse/HDFS-5178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771445#comment-13771445 ] Arpit Agarwal commented on HDFS-5178: - Alternatively you can take a look at the preliminary patch I attached to HDFS-4988 to check for potential conflicts. :-) It will need to be rebased after Nicholas's checkin for HDFS-4990. I should have the rebased patch up by tomorrow. DataTransferProtocol changes for supporting mutiple storages per DN --- Key: HDFS-5178 URL: https://issues.apache.org/jira/browse/HDFS-5178 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: Heterogeneous Storage (HDFS-2832) Reporter: Junping Du Assignee: Junping Du Per discussion in HDFS-5157, DataTransferProtocol should be updated to add StorageID info in some methods, i.e. writeBlock(), replaceBlock(), etc. After that, BlockReceiver (sender also) and receiveBlock can operate on target storage with new parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5178) DataTransferProtocol changes for supporting mutiple storages per DN
[ https://issues.apache.org/jira/browse/HDFS-5178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771458#comment-13771458 ] Junping Du commented on HDFS-5178: -- Sure. Thanks for reminder. Arpit! DataTransferProtocol changes for supporting mutiple storages per DN --- Key: HDFS-5178 URL: https://issues.apache.org/jira/browse/HDFS-5178 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: Heterogeneous Storage (HDFS-2832) Reporter: Junping Du Assignee: Junping Du Per discussion in HDFS-5157, DataTransferProtocol should be updated to add StorageID info in some methods, i.e. writeBlock(), replaceBlock(), etc. After that, BlockReceiver (sender also) and receiveBlock can operate on target storage with new parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5155) Depricate API of getStorageID() and setStorageID() in DatanodeID as multiple storages
[ https://issues.apache.org/jira/browse/HDFS-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771460#comment-13771460 ] Junping Du commented on HDFS-5155: -- Hi Arpit, looks like HDFS-4988 already cover the work to update storageID in DN. Shall we mark this JIRA as duplicated? Depricate API of getStorageID() and setStorageID() in DatanodeID as multiple storages - Key: HDFS-5155 URL: https://issues.apache.org/jira/browse/HDFS-5155 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: Heterogeneous Storage (HDFS-2832) Reporter: Junping Du Assignee: Junping Du StorageID is an unique number that associates the Datanodes blocks with a particular Datanode. To enable heterogeneous storage, we are supporting multiple storageIDs per DN, so previous APIs in DatanodeID, like: getStorageID() and setStorageID(), need to be updated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5155) Depricate API of getStorageID() and setStorageID() in DatanodeID as multiple storages
[ https://issues.apache.org/jira/browse/HDFS-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771470#comment-13771470 ] Arpit Agarwal commented on HDFS-5155: - Sounds good, I was just about to post the same. Depricate API of getStorageID() and setStorageID() in DatanodeID as multiple storages - Key: HDFS-5155 URL: https://issues.apache.org/jira/browse/HDFS-5155 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: Heterogeneous Storage (HDFS-2832) Reporter: Junping Du Assignee: Junping Du StorageID is an unique number that associates the Datanodes blocks with a particular Datanode. To enable heterogeneous storage, we are supporting multiple storageIDs per DN, so previous APIs in DatanodeID, like: getStorageID() and setStorageID(), need to be updated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5227) Namenode.getAddress() should return namenode rpc-address
[
https://issues.apache.org/jira/browse/HDFS-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771486#comment-13771486
]
Hadoop QA commented on HDFS-5227:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603934/HDFS-5227-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
org.apache.hadoop.hdfs.TestDFSShell
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4999//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4999//console
This message is automatically generated.
Namenode.getAddress() should return namenode rpc-address
Key: HDFS-5227
URL: https://issues.apache.org/jira/browse/HDFS-5227
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HDFS-5227-trunk.patch
Currently, {{Namenode.getAddress(Configuration conf)}} will return default
file system address as its result. The correct behavior should be returning
config value of dfs.namenode.rpc-address if it presents in the
configurations. Otherwise namenode will fail to start if the default file
system is configured to another file system other than the one running in the
cluster. We have a similar issue in 1.0 code base. The JIRA is HDFS-4320. The
previous JIRA is closed and we cannot open it. Thus create a new one to track
the issue in trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-5155) Depricate API of getStorageID() and setStorageID() in DatanodeID as multiple storages
[ https://issues.apache.org/jira/browse/HDFS-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du resolved HDFS-5155. -- Resolution: Duplicate Depricate API of getStorageID() and setStorageID() in DatanodeID as multiple storages - Key: HDFS-5155 URL: https://issues.apache.org/jira/browse/HDFS-5155 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: Heterogeneous Storage (HDFS-2832) Reporter: Junping Du Assignee: Junping Du StorageID is an unique number that associates the Datanodes blocks with a particular Datanode. To enable heterogeneous storage, we are supporting multiple storageIDs per DN, so previous APIs in DatanodeID, like: getStorageID() and setStorageID(), need to be updated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5167) Add metrics about the NameNode retry cache
[
https://issues.apache.org/jira/browse/HDFS-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771495#comment-13771495
]
Hadoop QA commented on HDFS-5167:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603937/HDFS-5167.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade
org.apache.hadoop.hdfs.server.namenode.TestSecondaryWebUi
org.apache.hadoop.hdfs.TestHDFSServerPorts
org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
org.apache.hadoop.hdfs.server.namenode.TestStartup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4998//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4998//console
This message is automatically generated.
Add metrics about the NameNode retry cache
--
Key: HDFS-5167
URL: https://issues.apache.org/jira/browse/HDFS-5167
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha, namenode
Affects Versions: 3.0.0
Reporter: Jing Zhao
Priority: Minor
Attachments: HDFS-5167.1.patch, HDFS-5167.2.patch, HDFS-5167.3.patch,
HDFS-5167.4.patch
It will be helpful to have metrics in NameNode about the retry cache, such as
the retry count etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over again
[
https://issues.apache.org/jira/browse/HDFS-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771499#comment-13771499
]
Colin Patrick McCabe commented on HDFS-5225:
It's not a log4j problem. The problem is that the DataNode is logging millions
of identical lines about the same block. The scanner has gotten stuck in a
loop, basically
datanode keeps logging the same 'is no longer in the dataset' message over
and over again
-
Key: HDFS-5225
URL: https://issues.apache.org/jira/browse/HDFS-5225
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.1.1-beta
Reporter: Roman Shaposhnik
I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
configuration: 4 nodes fully distributed cluster with security on.
All of a sudden my DN ate up all of the space in /var/log logging the
following message repeatedly:
{noformat}
2013-09-18 20:51:12,046 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer
in the dataset
{noformat}
It wouldn't answer to a jstack and jstack -F ended up being useless.
Here's what I was able to find in the NameNode logs regarding this block ID:
{noformat}
fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
BP-1884637155-10.224.158.152-1379524544853
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.224.158.152:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.34.74.206:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.83.107.80:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,899 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:17,904 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
2013-09-18 18:03:18,540 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:18,548 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1371
2013-09-18 18:03:26,150 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073742189_1371 10.83.107.80:1004 10.34.74.206:1004 10.224.158.152:1004
2013-09-18 18:03:26,847 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.34.74.206:1004 to delete [blk_1073742178_1359,
blk_1073742183_1362, blk_1073742184_1363, blk_1073742186_1366,
blk_1073742188_1368, blk_1073742189_1371]
2013-09-18 18:03:29,848 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.224.158.152:1004 to delete [blk_1073742177_1353,
blk_1073742178_1359, blk_1073742179_1355, blk_1073742180_1356,
[jira] [Commented] (HDFS-5208) Only clear network location cache on specific nodes if invalid NetworkTopology happens
[
https://issues.apache.org/jira/browse/HDFS-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771504#comment-13771504
]
Colin Patrick McCabe commented on HDFS-5208:
Unfortunately, {{DatanodeID#getHostName}} doesn't actually return the hostname.
It returns {{DatanodeID#hostName}}, which is either the registration name (if
it was specified) or the hostname (if it was not.)
I think this feature is mainly used in unit tests to fake up having different
hostnames-- something we could probably do this in a much simpler way. We've
discussed creating a JIRA to remove it before-- maybe it's time.
Only clear network location cache on specific nodes if invalid
NetworkTopology happens
--
Key: HDFS-5208
URL: https://issues.apache.org/jira/browse/HDFS-5208
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Junping Du
Assignee: Junping Du
Attachments: HDFS-5208-v1.patch
After HDFS-4521, once a DN is registered with invalid networktopology, all
cached rack info in DNSToSwitchMapping will be cleared. We should only clear
cache on specific nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4971) Move IO operations out of locking in OpenFileCtx
[ https://issues.apache.org/jira/browse/HDFS-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4971: Attachment: HDFS-4971.005.patch Update the patch after offline discussion with [~brandonli]. The main change includes: # Add processing logic for overlapped write # Handling closed FSDataOutputStream # Start dumper thread only when necessary # Initialize nextOffset Move IO operations out of locking in OpenFileCtx Key: HDFS-4971 URL: https://issues.apache.org/jira/browse/HDFS-4971 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-4971.000.patch, HDFS-4971.001.patch, HDFS-4971.002.patch, HDFS-4971.003.patch, HDFS-4971.004.patch, HDFS-4971.005.patch Currently some IO operations (such as writing data to HDFS and dumping to local disk) in OpenFileCtx may hold a lock which can block processing incoming writing requests. This jira aims to optimize OpenFileCtx and move the IO operations out of the locking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5137) Improve WriteManager for processing stable write requests and commit requests
[ https://issues.apache.org/jira/browse/HDFS-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5137: Attachment: HDFS-5137.patch Initial patch which depends on HDFS-4971. Improve WriteManager for processing stable write requests and commit requests - Key: HDFS-5137 URL: https://issues.apache.org/jira/browse/HDFS-5137 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Affects Versions: 3.0.0 Reporter: Jing Zhao Assignee: Jing Zhao Priority: Minor Attachments: HDFS-5137.patch WriteManager#handleCommit needs to block until all the data before the commitOffset has been received. This jira plans to improve the current concurrency implementation for this blocking call. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
Tsz Wo (Nicholas), SZE created HDFS-5228: Summary: The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE Key: HDFS-5228 URL: https://issues.apache.org/jira/browse/HDFS-5228 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative path. Then, it will result a NullPointerException when calling hasNext() from the RemoteIterator. This bug was discovered by Arnaud: http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[
https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-5228:
-
Attachment: h5228_20130919_test.patch
h5228_20130919_test.patch: a unit test for showing the bug. It will result the
following exception.
{noformat}
Running org.apache.hadoop.hdfs.TestDistributedFileSystem
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.991 sec
FAILURE! - in org.apache.hadoop.hdfs.TestDistributedFileSystem
testlistFiles(org.apache.hadoop.hdfs.TestDistributedFileSystem) Time elapsed:
2.839 sec ERROR!
java.lang.NullPointerException: null
at org.apache.hadoop.fs.Path.init(Path.java:105)
at org.apache.hadoop.fs.Path.makeQualified(Path.java:433)
at
org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus.makeQualifiedLocated(HdfsLocatedFileStatus.java:77)
at
org.apache.hadoop.hdfs.DistributedFileSystem$15.hasNext(DistributedFileSystem.java:739)
at org.apache.hadoop.fs.FileSystem$5.hasNext(FileSystem.java:1729)
at
org.apache.hadoop.hdfs.TestDistributedFileSystem.testlistFiles(TestDistributedFileSystem.java:855)
{noformat}
The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw
NPE
Key: HDFS-5228
URL: https://issues.apache.org/jira/browse/HDFS-5228
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Attachments: h5228_20130919_test.patch
Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative
path. Then, it will result a NullPointerException when calling hasNext()
from the RemoteIterator.
This bug was discovered by Arnaud:
http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over again
[
https://issues.apache.org/jira/browse/HDFS-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771520#comment-13771520
]
Junping Du commented on HDFS-5225:
--
Hi Colin and Roman, I think the following code won't delete block from
blockInfoSet if block is already removed from blockMap:
{code}
/** Deletes the block from internal structures */
synchronized void deleteBlock(Block block) {
BlockScanInfo info = blockMap.get(block);
if ( info != null ) {
delBlockInfo(info);
}
}
{code}
Then, if the block is happened to be the first block on blockInfoSet, then log
will loop forever.
The reason of inconsistent between blockMap and blockInfoSet may because
blockInfoSet is an unsynchronized TreeSet so is not thread-safe. May be we
should replace it with ConcurrentSkipListMap (which keep order and
concurrency). Thoughts?
datanode keeps logging the same 'is no longer in the dataset' message over
and over again
-
Key: HDFS-5225
URL: https://issues.apache.org/jira/browse/HDFS-5225
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.1.1-beta
Reporter: Roman Shaposhnik
I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
configuration: 4 nodes fully distributed cluster with security on.
All of a sudden my DN ate up all of the space in /var/log logging the
following message repeatedly:
{noformat}
2013-09-18 20:51:12,046 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer
in the dataset
{noformat}
It wouldn't answer to a jstack and jstack -F ended up being useless.
Here's what I was able to find in the NameNode logs regarding this block ID:
{noformat}
fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
BP-1884637155-10.224.158.152-1379524544853
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.224.158.152:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.34.74.206:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.83.107.80:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,899 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:17,904 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
2013-09-18 18:03:18,540 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:18,548 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1371
2013-09-18 18:03:26,150 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073742189_1371 10.83.107.80:1004 10.34.74.206:1004
[jira] [Commented] (HDFS-4971) Move IO operations out of locking in OpenFileCtx
[
https://issues.apache.org/jira/browse/HDFS-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771524#comment-13771524
]
Hadoop QA commented on HDFS-4971:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603970/HDFS-4971.005.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5000//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5000//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs-nfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5000//console
This message is automatically generated.
Move IO operations out of locking in OpenFileCtx
Key: HDFS-4971
URL: https://issues.apache.org/jira/browse/HDFS-4971
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-4971.000.patch, HDFS-4971.001.patch,
HDFS-4971.002.patch, HDFS-4971.003.patch, HDFS-4971.004.patch,
HDFS-4971.005.patch
Currently some IO operations (such as writing data to HDFS and dumping to
local disk) in OpenFileCtx may hold a lock which can block processing
incoming writing requests. This jira aims to optimize OpenFileCtx and move
the IO operations out of the locking.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[ https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-5228: - Attachment: h5228_20130919.patch h5228_20130919.patch: makeQualified if the path is a relative path. The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE Key: HDFS-5228 URL: https://issues.apache.org/jira/browse/HDFS-5228 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h5228_20130919.patch, h5228_20130919_test.patch Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative path. Then, it will result a NullPointerException when calling hasNext() from the RemoteIterator. This bug was discovered by Arnaud: http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5208) Only clear network location cache on specific nodes if invalid NetworkTopology happens
[ https://issues.apache.org/jira/browse/HDFS-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771529#comment-13771529 ] Junping Du commented on HDFS-5208: -- Thanks for comments, Colin! bq. Unfortunately, DatanodeID#getHostName doesn't actually return the hostname. It returns DatanodeID#hostName, which is either the registration name (if it was specified) or the hostname (if it was not.) So, we are saying the same thing - DatanodeID is created in DN heartbeat to NN for registration and its hostName comes from conf of dfs.datanode.hostname which can be any style but DNS name if this config is not setting. - Isn't it? :) bq. I think this feature is mainly used in unit tests to fake up having different hostnames-- something we could probably do this in a much simpler way. We've discussed creating a JIRA to remove it before-- maybe it's time. I agree. Will file separated Jira and work on it later. So we may just do simply two things below: 1. remove config of dfs.datanode.hostname and all usage place. 2. make sure DatanodeID#hostname is its hostname (DNS name). Anything else to address? Only clear network location cache on specific nodes if invalid NetworkTopology happens -- Key: HDFS-5208 URL: https://issues.apache.org/jira/browse/HDFS-5208 Project: Hadoop HDFS Issue Type: Improvement Reporter: Junping Du Assignee: Junping Du Attachments: HDFS-5208-v1.patch After HDFS-4521, once a DN is registered with invalid networktopology, all cached rack info in DNSToSwitchMapping will be cleared. We should only clear cache on specific nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5191) revisit zero-copy API in FSDataInputStream to make it more intuitive
[ https://issues.apache.org/jira/browse/HDFS-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5191: --- Attachment: HDFS-5191-caching.003.patch * revise libhdfs bindings * avoid need for wrappers around ByteBuffers when storing them revisit zero-copy API in FSDataInputStream to make it more intuitive Key: HDFS-5191 URL: https://issues.apache.org/jira/browse/HDFS-5191 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client, libhdfs Affects Versions: HDFS-4949 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-5191-caching.001.patch, HDFS-5191-caching.003.patch As per the discussion on HDFS-4953, we should revisit the zero-copy API to make it more intuitive for new users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[ https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-5228: - Status: Patch Available (was: Open) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE Key: HDFS-5228 URL: https://issues.apache.org/jira/browse/HDFS-5228 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h5228_20130919.patch, h5228_20130919_test.patch Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative path. Then, it will result a NullPointerException when calling hasNext() from the RemoteIterator. This bug was discovered by Arnaud: http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over again
[
https://issues.apache.org/jira/browse/HDFS-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771538#comment-13771538
]
Tsuyoshi OZAWA commented on HDFS-5225:
--
Colin and Roman,
thank you for sharing! I understood the problem correctly.
Junping,
Both blockMap and blockInfoSet must be synchronized by BlockPoolSliceScanner's
instance in this case. verifyFirstBlock method refers blockInfoSet, but it's
not synchronized. IMHO, if we can make verifyFirstBlock method synchronized,
this problem may be solved. What do you think?
datanode keeps logging the same 'is no longer in the dataset' message over
and over again
-
Key: HDFS-5225
URL: https://issues.apache.org/jira/browse/HDFS-5225
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.1.1-beta
Reporter: Roman Shaposhnik
I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
configuration: 4 nodes fully distributed cluster with security on.
All of a sudden my DN ate up all of the space in /var/log logging the
following message repeatedly:
{noformat}
2013-09-18 20:51:12,046 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer
in the dataset
{noformat}
It wouldn't answer to a jstack and jstack -F ended up being useless.
Here's what I was able to find in the NameNode logs regarding this block ID:
{noformat}
fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
BP-1884637155-10.224.158.152-1379524544853
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.224.158.152:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.34.74.206:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.83.107.80:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,899 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:17,904 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
2013-09-18 18:03:18,540 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:18,548 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1371
2013-09-18 18:03:26,150 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073742189_1371 10.83.107.80:1004 10.34.74.206:1004 10.224.158.152:1004
2013-09-18 18:03:26,847 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.34.74.206:1004 to delete [blk_1073742178_1359,
blk_1073742183_1362, blk_1073742184_1363, blk_1073742186_1366,
blk_1073742188_1368, blk_1073742189_1371]
2013-09-18 18:03:29,848 INFO
[jira] [Commented] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over again
[
https://issues.apache.org/jira/browse/HDFS-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771542#comment-13771542
]
Tsuyoshi OZAWA commented on HDFS-5225:
--
verifyFirstBlock method refers blockInfoSet, but it's not synchronized.
IMHO, if we can make verifyFirstBlock method synchronized, this problem may
be solved. What do you think?
Sorry, I meant not verifyFirstBlock method, but scan method.
datanode keeps logging the same 'is no longer in the dataset' message over
and over again
-
Key: HDFS-5225
URL: https://issues.apache.org/jira/browse/HDFS-5225
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.1.1-beta
Reporter: Roman Shaposhnik
I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
configuration: 4 nodes fully distributed cluster with security on.
All of a sudden my DN ate up all of the space in /var/log logging the
following message repeatedly:
{noformat}
2013-09-18 20:51:12,046 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer
in the dataset
{noformat}
It wouldn't answer to a jstack and jstack -F ended up being useless.
Here's what I was able to find in the NameNode logs regarding this block ID:
{noformat}
fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
BP-1884637155-10.224.158.152-1379524544853
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.224.158.152:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.34.74.206:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.83.107.80:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,899 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:17,904 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
2013-09-18 18:03:18,540 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:18,548 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1371
2013-09-18 18:03:26,150 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073742189_1371 10.83.107.80:1004 10.34.74.206:1004 10.224.158.152:1004
2013-09-18 18:03:26,847 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.34.74.206:1004 to delete [blk_1073742178_1359,
blk_1073742183_1362, blk_1073742184_1363, blk_1073742186_1366,
blk_1073742188_1368, blk_1073742189_1371]
2013-09-18 18:03:29,848 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.224.158.152:1004 to delete [blk_1073742177_1353,
[jira] [Updated] (HDFS-5225) datanode keeps logging the same 'is no longer in the dataset' message over and over again
[
https://issues.apache.org/jira/browse/HDFS-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi OZAWA updated HDFS-5225:
-
Attachment: HDFS-5225.1.patch
Maybe this is a critical section to be fixed.
datanode keeps logging the same 'is no longer in the dataset' message over
and over again
-
Key: HDFS-5225
URL: https://issues.apache.org/jira/browse/HDFS-5225
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.1.1-beta
Reporter: Roman Shaposhnik
Attachments: HDFS-5225.1.patch
I was running the usual Bigtop testing on 2.1.1-beta RC1 with the following
configuration: 4 nodes fully distributed cluster with security on.
All of a sudden my DN ate up all of the space in /var/log logging the
following message repeatedly:
{noformat}
2013-09-18 20:51:12,046 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369 is no longer
in the dataset
{noformat}
It wouldn't answer to a jstack and jstack -F ended up being useless.
Here's what I was able to find in the NameNode logs regarding this block ID:
{noformat}
fgrep -rI 'blk_1073742189' hadoop-hdfs-namenode-ip-10-224-158-152.log
2013-09-18 18:03:16,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/user/jenkins/testAppendInputWedSep18180222UTC2013/test4.fileWedSep18180222UTC2013._COPYING_.
BP-1884637155-10.224.158.152-1379524544853
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]}
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.224.158.152:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,222 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.34.74.206:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,224 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.83.107.80:1004 is added to
blk_1073742189_1369{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.83.107.80:1004|RBW],
ReplicaUnderConstruction[10.34.74.206:1004|RBW],
ReplicaUnderConstruction[10.224.158.152:1004|RBW]]} size 0
2013-09-18 18:03:17,899 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369,
newGenerationStamp=1370, newLength=1048576, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:17,904 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1369)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370
2013-09-18 18:03:18,540 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(block=BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370,
newGenerationStamp=1371, newLength=2097152, newNodes=[10.83.107.80:1004,
10.34.74.206:1004, 10.224.158.152:1004],
clientName=DFSClient_NONMAPREDUCE_-450304083_1)
2013-09-18 18:03:18,548 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
updatePipeline(BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1370)
successfully to
BP-1884637155-10.224.158.152-1379524544853:blk_1073742189_1371
2013-09-18 18:03:26,150 INFO BlockStateChange: BLOCK* addToInvalidates:
blk_1073742189_1371 10.83.107.80:1004 10.34.74.206:1004 10.224.158.152:1004
2013-09-18 18:03:26,847 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.34.74.206:1004 to delete [blk_1073742178_1359,
blk_1073742183_1362, blk_1073742184_1363, blk_1073742186_1366,
blk_1073742188_1368, blk_1073742189_1371]
2013-09-18 18:03:29,848 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask 10.224.158.152:1004 to delete [blk_1073742177_1353,
blk_1073742178_1359, blk_1073742179_1355, blk_1073742180_1356,
blk_1073742181_1358, blk_1073742182_1361, blk_1073742185_1364,
blk_1073742187_1367, blk_1073742188_1368, blk_1073742189_1371]
[jira] [Updated] (HDFS-4971) Move IO operations out of locking in OpenFileCtx
[ https://issues.apache.org/jira/browse/HDFS-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4971: Attachment: HDFS-4971.006.patch Fix the findbug. Move IO operations out of locking in OpenFileCtx Key: HDFS-4971 URL: https://issues.apache.org/jira/browse/HDFS-4971 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-4971.000.patch, HDFS-4971.001.patch, HDFS-4971.002.patch, HDFS-4971.003.patch, HDFS-4971.004.patch, HDFS-4971.005.patch, HDFS-4971.006.patch Currently some IO operations (such as writing data to HDFS and dumping to local disk) in OpenFileCtx may hold a lock which can block processing incoming writing requests. This jira aims to optimize OpenFileCtx and move the IO operations out of the locking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4971) Move IO operations out of locking in OpenFileCtx
[
https://issues.apache.org/jira/browse/HDFS-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771576#comment-13771576
]
Hadoop QA commented on HDFS-4971:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603983/HDFS-4971.006.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5002//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5002//console
This message is automatically generated.
Move IO operations out of locking in OpenFileCtx
Key: HDFS-4971
URL: https://issues.apache.org/jira/browse/HDFS-4971
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-4971.000.patch, HDFS-4971.001.patch,
HDFS-4971.002.patch, HDFS-4971.003.patch, HDFS-4971.004.patch,
HDFS-4971.005.patch, HDFS-4971.006.patch
Currently some IO operations (such as writing data to HDFS and dumping to
local disk) in OpenFileCtx may hold a lock which can block processing
incoming writing requests. This jira aims to optimize OpenFileCtx and move
the IO operations out of the locking.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5031) BlockScanner scans the block multiple times and on restart scans everything
[ https://issues.apache.org/jira/browse/HDFS-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771579#comment-13771579 ] Vinay commented on HDFS-5031: - Thanks Arpit for reviews and commit. BlockScanner scans the block multiple times and on restart scans everything --- Key: HDFS-5031 URL: https://issues.apache.org/jira/browse/HDFS-5031 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0, 2.1.0-beta Reporter: Vinay Assignee: Vinay Fix For: 2.3.0 Attachments: HDFS-5031.patch, HDFS-5031.patch, HDFS-5031.patch BlockScanner scans the block twice, also on restart of datanode scans everything. Steps: 1. Write blocks with interval of more than 5 seconds. write new block on completion of scan for written block. Each time datanode scans new block, it also scans, previous block which is already scanned. Now after restart, datanode scans all blocks again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[
https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771581#comment-13771581
]
Hadoop QA commented on HDFS-5228:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603973/h5228_20130919.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDFSShell
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5001//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5001//console
This message is automatically generated.
The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw
NPE
Key: HDFS-5228
URL: https://issues.apache.org/jira/browse/HDFS-5228
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Attachments: h5228_20130919.patch, h5228_20130919_test.patch
Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative
path. Then, it will result a NullPointerException when calling hasNext()
from the RemoteIterator.
This bug was discovered by Arnaud:
http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5228) The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE
[ https://issues.apache.org/jira/browse/HDFS-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771582#comment-13771582 ] Tsz Wo (Nicholas), SZE commented on HDFS-5228: -- The test failure does not seem related. It also failed in some previous build (e.g. [build #4999|https://builds.apache.org/job/PreCommit-HDFS-Build/4999/testReport/org.apache.hadoop.hdfs/TestDFSShell/testCopyCommandsWithForceOption/]) and my local machine without the patch. The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE Key: HDFS-5228 URL: https://issues.apache.org/jira/browse/HDFS-5228 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h5228_20130919.patch, h5228_20130919_test.patch Get a RemoteIterator from DistributedFileSystem.listFiles(..) with a relative path. Then, it will result a NullPointerException when calling hasNext() from the RemoteIterator. This bug was discovered by Arnaud: http://hortonworks.com/community/forums/topic/new-bug-in-hdfs-listfiles-method/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: HDFS-4988.02.patch Rebased patch. Test cases will need to be fixed. Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
