[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[
https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364100#comment-14364100
]
Jing Zhao commented on HDFS-7827:
-
Currently we still have non-protobuf image for offline processing. Thus this
change can still be helpful especially for storage usage analysis.
Some comments on the current patch:
# After updating {{writeINodeUnderConstruction}}, we also need to update
{{readINodeUnderConstruction}} accordingly.
# We also need a unit test to cover the under construction file with striped
blocks.
# Instead of the following code, we can simply use
out.writeBoolean(cons.isWithStripedBlocks()). Whether we keep using
EC_STORAGE_POLICY_ID is still under construction.
{code}
+if (cons.isStriped()){
+ out.writeByte(HdfsConstants.EC_STORAGE_POLICY_ID);
+}else{
+ out.writeByte(0);
+}
{code}
# Please correct the coding format according to the project conventions.
Besides, as [~wheat9] pointed out, we also need to update WebImageViewer for
file size calculation. Please open a separate jira to do this.
Erasure Coding: support striped blocks in non-protobuf fsimage
--
Key: HDFS-7827
URL: https://issues.apache.org/jira/browse/HDFS-7827
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Hui Zheng
Attachments: HDFS-7827.000.patch, HDFS-7827.001.patch,
HDFS-7827.002.patch
HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage.
We should also add this support to the non-protobuf fsimage since it is still
used for use cases like offline image processing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7937) Erasure Coding: INodeFile quota computation unit tests
Kai Sasaki created HDFS-7937: Summary: Erasure Coding: INodeFile quota computation unit tests Key: HDFS-7937 URL: https://issues.apache.org/jira/browse/HDFS-7937 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Sasaki Assignee: Kai Sasaki Priority: Minor Unit test for [HDFS-7826|https://issues.apache.org/jira/browse/HDFS-7826] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7854) Separate class DataStreamer out of DFSOutputStream
[
https://issues.apache.org/jira/browse/HDFS-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364386#comment-14364386
]
Hadoop QA commented on HDFS-7854:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704951/HDFS-7854-004-duplicate3.patch
against trunk revision 5608520.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9908//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9908//console
This message is automatically generated.
Separate class DataStreamer out of DFSOutputStream
--
Key: HDFS-7854
URL: https://issues.apache.org/jira/browse/HDFS-7854
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Li Bo
Assignee: Li Bo
Attachments: HDFS-7854-001.patch, HDFS-7854-002.patch,
HDFS-7854-003.patch, HDFS-7854-004-duplicate.patch,
HDFS-7854-004-duplicate2.patch, HDFS-7854-004-duplicate3.patch,
HDFS-7854-004.patch
This sub task separate DataStreamer from DFSOutputStream. New DataStreamer
will accept packets and write them to remote datanodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7938) OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit Mac
Colin Patrick McCabe created HDFS-7938: -- Summary: OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit Mac Key: HDFS-7938 URL: https://issues.apache.org/jira/browse/HDFS-7938 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Colin Patrick McCabe Priority: Critical In OpensslSecureRandom.c, pthread_threadid_np is being used with an unsigned long, but the type signature requires a uint64_t. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7935) Support multi-homed networks when Kerberos security is enabled
[ https://issues.apache.org/jira/browse/HDFS-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363895#comment-14363895 ] Kihwal Lee commented on HDFS-7935: -- Secure HA NN does come up with dfs.namenode.http-address.logical_name.nn_id set to 0.0.0.0:port. The clients should of course have a vaild address. The kerberos principal used by the two name nodes comes from dfs.web.authentication.kerberos.principal, but SPNEGO will work with any SPN found in its keytab after HADOOP-10322. So I guess HDFS-4448 is fixed? Support multi-homed networks when Kerberos security is enabled -- Key: HDFS-7935 URL: https://issues.apache.org/jira/browse/HDFS-7935 Project: Hadoop HDFS Issue Type: Bug Reporter: Arun Suresh Assignee: Arun Suresh Currently, during SASL negotiation stage between ipc Client and Server, The server sends only a single serviceId (curresponding to a single principal) to the client. This is the principal the the server process is logged in as during startup. It is possible that in a multi-homed network, the server might be associated with more than one principal, and thus severs must provide the clients all possible principals it can use to connect to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-2743) Streamline usage of bookkeeper journal manager
[ https://issues.apache.org/jira/browse/HDFS-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-2743: --- Fix Version/s: (was: 2.0.0-alpha) 3.0.0 Streamline usage of bookkeeper journal manager -- Key: HDFS-2743 URL: https://issues.apache.org/jira/browse/HDFS-2743 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: 3.0.0 Attachments: HDFS-2743.diff, HDFS-2743.diff The current method of installing bkjournal manager involves generating a tarball, and extracting it with special flags over the hdfs distribution. This is cumbersome and prone to being broken by other changes (see https://svn.apache.org/repos/asf/hadoop/common/trunk@1220940). I think a cleaner way to doing this is to generate a single jar that can be placed in the lib dir of hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7936) Erasure coding: resolving conflicts when merging with HDFS-7903 and HDFS-7435
[ https://issues.apache.org/jira/browse/HDFS-7936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-7936: Attachment: HDFS-7936-002.patch 001 patch had unnecessary changes to import Erasure coding: resolving conflicts when merging with HDFS-7903 and HDFS-7435 - Key: HDFS-7936 URL: https://issues.apache.org/jira/browse/HDFS-7936 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Jing Zhao Attachments: HDFS-7936-001.patch, HDFS-7936-002.patch A few non-trivial conflicts were found when merging trunk changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7936) Erasure coding: resolving conflicts when merging with HDFS-7903 and HDFS-7435
[ https://issues.apache.org/jira/browse/HDFS-7936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363961#comment-14363961 ] Zhe Zhang commented on HDFS-7936: - [~jingzhao] Thanks! I actually have a patch (guess JIRA has a delay in showing it). Let me know if the 002 patch looks correct to you. Erasure coding: resolving conflicts when merging with HDFS-7903 and HDFS-7435 - Key: HDFS-7936 URL: https://issues.apache.org/jira/browse/HDFS-7936 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Jing Zhao Attachments: HDFS-7936-001.patch, HDFS-7936-002.patch A few non-trivial conflicts were found when merging trunk changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7935) Support multi-homed networks when Kerberos security is enabled
[ https://issues.apache.org/jira/browse/HDFS-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363979#comment-14363979 ] Arun Suresh commented on HDFS-7935: --- Hit enter by mistake... Continuing.. So looks like we still need the HDFS-4448 fix (I guess it needs a minor rebasing though..) Support multi-homed networks when Kerberos security is enabled -- Key: HDFS-7935 URL: https://issues.apache.org/jira/browse/HDFS-7935 Project: Hadoop HDFS Issue Type: Bug Reporter: Arun Suresh Assignee: Arun Suresh Currently, during SASL negotiation stage between ipc Client and Server, The server sends only a single serviceId (curresponding to a single principal) to the client. This is the principal the the server process is logged in as during startup. It is possible that in a multi-homed network, the server might be associated with more than one principal, and thus severs must provide the clients all possible principals it can use to connect to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-3265) PowerPc Build error.
[ https://issues.apache.org/jira/browse/HDFS-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364042#comment-14364042 ] Allen Wittenauer commented on HDFS-3265: For those spelunking: this was never committed to any other branches other than 1.x and trunk. PowerPc Build error. Key: HDFS-3265 URL: https://issues.apache.org/jira/browse/HDFS-3265 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 1.0.2, 1.0.3, 2.0.0-alpha Environment: Linux RHEL 6.1 PowerPC + IBM JVM 6.0 SR10 Reporter: Kumar Ravi Assignee: Kumar Ravi Labels: patch Fix For: 1.0.3, 3.0.0 Attachments: HADOOP-8271.patch, HADOOP-8271.patch Original Estimate: 168h Remaining Estimate: 168h When attempting to build branch-1, the following error is seen and ant exits. [exec] configure: error: Unsupported CPU architecture powerpc64 The following command was used to build hadoop-common ant -Dlibhdfs=true -Dcompile.native=true -Dfusedfs=true -Dcompile.c++=true -Dforrest.home=$FORREST_HOME compile-core-native compile-c++ compile-c++-examples task-controller tar record-parser compile-hdfs-classes package -Djava5.home=/opt/ibm/ibm-java2-ppc64-50/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-7933) fsck should also report decommissioning replicas.
[ https://issues.apache.org/jira/browse/HDFS-7933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao reassigned HDFS-7933: Assignee: Xiaoyu Yao fsck should also report decommissioning replicas. -- Key: HDFS-7933 URL: https://issues.apache.org/jira/browse/HDFS-7933 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Jitendra Nath Pandey Assignee: Xiaoyu Yao Fsck doesn't count replicas that are on decommissioning nodes. If a block has all replicas on the decommissioning nodes, it will be marked as missing, which is alarming for the admins, although the system will replicate them before nodes are decommissioned. Fsck output should also show decommissioning replicas along with the live replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3549) dist tar build fails in hadoop-hdfs-raid project
[
https://issues.apache.org/jira/browse/HDFS-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HDFS-3549:
---
Fix Version/s: (was: 2.0.0-alpha)
3.0.0
dist tar build fails in hadoop-hdfs-raid project
Key: HDFS-3549
URL: https://issues.apache.org/jira/browse/HDFS-3549
Project: Hadoop HDFS
Issue Type: Bug
Components: build
Affects Versions: 3.0.0
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical
Fix For: 3.0.0
Attachments: HDFS-3549.patch, HDFS-3549.patch, HDFS-3549.patch,
HDFS-3549.patch
Trying to build the distribution tarball in a clean tree via {{mvn install
-Pdist -Dtar -DskipTests -Dmaven.javadoc.skip}} fails with this error:
{noformat}
main:
[exec] tar: hadoop-hdfs-raid-3.0.0-SNAPSHOT: Cannot stat: No such file
or directory
[exec] tar: Exiting with failure status due to previous errors
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7878) API - expose an unique file identifier
[ https://issues.apache.org/jira/browse/HDFS-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364104#comment-14364104 ] Sergey Shelukhin commented on HDFS-7878: Note: confusingly, HdfsFileStatus is not FileStatus, that's an internal class [~cmccabe] can you clarify what you mean by two RPCs? The API makes one RPC. Two RPCs will only happen if the user calls both get ID and get status, which is not necessary. API - expose an unique file identifier -- Key: HDFS-7878 URL: https://issues.apache.org/jira/browse/HDFS-7878 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HDFS-7878.01.patch, HDFS-7878.02.patch, HDFS-7878.patch See HDFS-487. Even though that is resolved as duplicate, the ID is actually not exposed by the JIRA it supposedly duplicates. INode ID for the file should be easy to expose; alternatively ID could be derived from block IDs, to account for appends... This is useful e.g. for cache key by file, to make sure cache stays correct when file is overwritten. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7826) Erasure Coding: Update INodeFile quota computation for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364218#comment-14364218
]
Jing Zhao commented on HDFS-7826:
-
+1 for the .5 patch. I will commit it shortly.
Erasure Coding: Update INodeFile quota computation for striped blocks
-
Key: HDFS-7826
URL: https://issues.apache.org/jira/browse/HDFS-7826
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Kai Sasaki
Attachments: HDFS-7826.1.patch, HDFS-7826.2.patch, HDFS-7826.3.patch,
HDFS-7826.4.patch, HDFS-7826.5.patch
Currently INodeFile's quota computation only considers contiguous blocks
(i.e., {{INodeFile#blocks}}). We need to update it to support striped blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7054) Make DFSOutputStream tracing more fine-grained
[ https://issues.apache.org/jira/browse/HDFS-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-7054: --- Attachment: HDFS-7054.001.patch Make DFSOutputStream tracing more fine-grained -- Key: HDFS-7054 URL: https://issues.apache.org/jira/browse/HDFS-7054 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 2.6.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7054.001.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7054) Make DFSOutputStream tracing more fine-grained
[ https://issues.apache.org/jira/browse/HDFS-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-7054: --- Target Version/s: 2.7.0 (was: 2.6.0) Status: Patch Available (was: Open) Make DFSOutputStream tracing more fine-grained -- Key: HDFS-7054 URL: https://issues.apache.org/jira/browse/HDFS-7054 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 2.6.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7054.001.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7854) Separate class DataStreamer out of DFSOutputStream
[ https://issues.apache.org/jira/browse/HDFS-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-7854: Attachment: HDFS-7854-004-duplicate3.patch The findbugsWarnings's url cannot be opened, trigger Jenkins again. Separate class DataStreamer out of DFSOutputStream -- Key: HDFS-7854 URL: https://issues.apache.org/jira/browse/HDFS-7854 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Li Bo Assignee: Li Bo Attachments: HDFS-7854-001.patch, HDFS-7854-002.patch, HDFS-7854-003.patch, HDFS-7854-004-duplicate.patch, HDFS-7854-004-duplicate2.patch, HDFS-7854-004-duplicate3.patch, HDFS-7854-004.patch This sub task separate DataStreamer from DFSOutputStream. New DataStreamer will accept packets and write them to remote datanodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7912) Erasure Coding: track BlockInfo instead of Block in UnderReplicatedBlocks and PendingReplicationBlocks
[
https://issues.apache.org/jira/browse/HDFS-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364346#comment-14364346
]
Zhe Zhang commented on HDFS-7912:
-
Thanks Jing for clarifying about NN overhead.
About the second minor point, isn't {{storedBlock}} always equal to {{lBlk}}
earlier in the code?
{code}
for (LocatedBlock lBlk : blocks.getLocatedBlocks()) {
ExtendedBlock block = lBlk.getBlock();
{code}
Erasure Coding: track BlockInfo instead of Block in UnderReplicatedBlocks and
PendingReplicationBlocks
--
Key: HDFS-7912
URL: https://issues.apache.org/jira/browse/HDFS-7912
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-7912.000.patch
Now with striped blocks and the design that uses a single BlockInfoStriped
object to track all the corresponding blocks, we need to clearly distinguish
the type Block and BlockInfo in BlockManager. Specifically, data structures
like {{UnderReplicatedBlocks}} and {{PendingReplicationBlocks}} should track
BlockInfo instead of Block in order to support striped block recovery.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7878) API - expose an unique file identifier
[
https://issues.apache.org/jira/browse/HDFS-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364389#comment-14364389
]
Colin Patrick McCabe commented on HDFS-7878:
Hi [~jingzhao],
I understand that the current patch only makes one RPC. My concern is that in
order to get both the file ID and the other information about the file, two
RPCs will be needed when using this API. It's just a bad API from an
efficiency and concurrency point of view. It would be like if you needed 5
separate RPCs to get the length, access time, modification time, group, and
user. There is a reason why FileStatus combines together all these fields into
one class, and that same logic should lead us to including a way of getting
file ID out of FileStatus.
I understand that HdfsFileStatus already has {{HdfsFileStatus#getFileId}}. But
this functionality should not be HDFS-specific. There are other filesystems
that have file IDs. (Plus, if casting to HdfsFileStatus was adequate, there
would be no need for this JIRA at all, since this cast is already possible.)
Why not just add an accessor function to {{FileStatus}} with a default
implementation like I suggested earlier? Adding a new function to a stable
interface is a compatible change (and I note, that adding a new function to
FileSystem is also adding a new function to a stable class).
API - expose an unique file identifier
--
Key: HDFS-7878
URL: https://issues.apache.org/jira/browse/HDFS-7878
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HDFS-7878.01.patch, HDFS-7878.02.patch, HDFS-7878.patch
See HDFS-487.
Even though that is resolved as duplicate, the ID is actually not exposed by
the JIRA it supposedly duplicates.
INode ID for the file should be easy to expose; alternatively ID could be
derived from block IDs, to account for appends...
This is useful e.g. for cache key by file, to make sure cache stays correct
when file is overwritten.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7937) Erasure Coding: INodeFile quota computation unit tests
[ https://issues.apache.org/jira/browse/HDFS-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HDFS-7937: - Attachment: (was: HDFS-7937.1.patch) Erasure Coding: INodeFile quota computation unit tests -- Key: HDFS-7937 URL: https://issues.apache.org/jira/browse/HDFS-7937 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Sasaki Assignee: Kai Sasaki Priority: Minor Unit test for [HDFS-7826|https://issues.apache.org/jira/browse/HDFS-7826] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7902) Expose truncate API for libwebhdfs
[
https://issues.apache.org/jira/browse/HDFS-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364457#comment-14364457
]
Colin Patrick McCabe commented on HDFS-7902:
{code}
130 int createUrlForNnTRUNCATE(const char *host, int nnPort, const char
*path,
131size_t newlength, const char *user, char
**url)
...
139 strlength = snprintf(newlengthStr, LONG_STR_LEN, %lu,
newlength);
{code}
newlength should be a {{tOffset}} like it is in the {{hdfs.h}} API. It
should be {{%llu}} because on 32-bit systems, long is not a 64-bit type.
{code}
1143ret = createUrlForNnTRUNCATE(fs-nn, fs-port, absPath,
1144 newlength, fs-userName, url);
1145free(absPath);
1146if (ret) {
1147errno = ret;
1148return -1;
1149}
{code}
url isn't freed on the error case here. Similarly with resp.
Expose truncate API for libwebhdfs
--
Key: HDFS-7902
URL: https://issues.apache.org/jira/browse/HDFS-7902
Project: Hadoop HDFS
Issue Type: Improvement
Components: native, webhdfs
Affects Versions: 2.7.0
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-7902.001.patch, HDFS-7902.002.patch
As Colin suggested in HDFS-7838, we will add truncate support for libwebhdfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7938) OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit Mac
[ https://issues.apache.org/jira/browse/HDFS-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364488#comment-14364488 ] Chris Nauroth commented on HDFS-7938: - Hello, [~kiranmr]. Are you interested in taking this as a follow-up? OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit Mac Key: HDFS-7938 URL: https://issues.apache.org/jira/browse/HDFS-7938 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Colin Patrick McCabe Priority: Critical In OpensslSecureRandom.c, pthread_threadid_np is being used with an unsigned long, but the type signature requires a uint64_t. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7937) Erasure Coding: INodeFile quota computation unit tests
[
https://issues.apache.org/jira/browse/HDFS-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364250#comment-14364250
]
Jing Zhao commented on HDFS-7937:
-
Thanks for opening the jira, Kai.
BTW, there is some code cleanup work we can do for the change brought by
HDFS-7826. Specifically, {{computeQuotaUsageWithStriped}} does not use the
parameter bsps. Also we need to fix its javadoc. You can include the cleanup in
this jira.
Erasure Coding: INodeFile quota computation unit tests
--
Key: HDFS-7937
URL: https://issues.apache.org/jira/browse/HDFS-7937
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Priority: Minor
Unit test for [HDFS-7826|https://issues.apache.org/jira/browse/HDFS-7826]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-2360) Ugly stacktrace when quota exceeds
[
https://issues.apache.org/jira/browse/HDFS-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364326#comment-14364326
]
Hadoop QA commented on HDFS-2360:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704900/HDFS-2360.patch
against trunk revision 2681ed9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestFileCreation
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9906//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9906//console
This message is automatically generated.
Ugly stacktrace when quota exceeds
--
Key: HDFS-2360
URL: https://issues.apache.org/jira/browse/HDFS-2360
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 0.23.0
Reporter: Rajit Saha
Assignee: Harsh J
Priority: Minor
Attachments: HDFS-2360.patch, HDFS-2360.patch
Will it be better to catch the exception and throw a small reasonable messege
to user when they exceed quota?
$hdfs dfs -mkdir testDir
$hdfs dfsadmin -setSpaceQuota 191M testDir
$hdfs dfs -count -q testDir
none inf 200278016 2002780161
0 0
hdfs://NN hostname:port/user/hdfsqa/testDir
$hdfs dfs -put /etc/passwd /user/hadoopqa/testDir
11/09/19 08:08:15 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota
of /user/hdfsqa/testDir is exceeded:
quota=191.0m diskspace consumed=768.0m
at
org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:159)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1609)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1383)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:370)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1681)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1476)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:389)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1496)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1492)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1490)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
at
[jira] [Commented] (HDFS-7054) Make DFSOutputStream tracing more fine-grained
[ https://issues.apache.org/jira/browse/HDFS-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364418#comment-14364418 ] Colin Patrick McCabe commented on HDFS-7054: v2 fixes a missing import Make DFSOutputStream tracing more fine-grained -- Key: HDFS-7054 URL: https://issues.apache.org/jira/browse/HDFS-7054 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 2.6.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7054.001.patch, HDFS-7054.002.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5523) Support multiple subdirectory exports in HDFS NFS gateway
[
https://issues.apache.org/jira/browse/HDFS-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364259#comment-14364259
]
Brandon Li commented on HDFS-5523:
--
{quote}We should probably disallow nesting in this phase.{quote}
Sounds ok to me. Otherwise, it would be hard to decide the access mode, for
example: export /a is read-only but /a/b is read-write, when user traverse from
/a to /a/b, it's tricky to decide which access the user should have.
{quote}I think we should allow mounting a non-top-level directory.{quote}
In this case, we can have root as the only mount point and the users can mount
any sub-directory. This might be the easiest way to implement, but provide only
marginal security provision. Users can mount any sub-directories as long as he
has the right to do mount operation. Usually a root user is needed to do mount,
so if customers' environment has control over who can do mount, then this
option is viable solution. Of course, to add a bit more security control, the
user can choose to export multiple sub-directories instead of only root.
Support multiple subdirectory exports in HDFS NFS gateway
--
Key: HDFS-5523
URL: https://issues.apache.org/jira/browse/HDFS-5523
Project: Hadoop HDFS
Issue Type: New Feature
Components: nfs
Reporter: Brandon Li
Currently, the HDFS NFS Gateway only supports configuring a single
subdirectory export via the {{dfs.nfs3.export.point}} configuration setting.
Supporting multiple subdirectory exports can make data and security
management easier when using the HDFS NFS Gateway.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7864) Erasure Coding: Update safemode calculation for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364289#comment-14364289
]
Jing Zhao commented on HDFS-7864:
-
Thanks for updating the patch, Rui! Some comments for the latest patch:
# Actually we do not need {{DFS_NAMENODE_STRIPE_MIN_DEFAULT}}. For a striped
block, the minimal safe number should depend on the EC schema of the
individual block (i.e., the number of data blocks of a striped block group).
Therefore we also need to update the logic of {{incrementSafeBlockCount}}.
# There are still multiple places not following the current project code
convention. E.g., we need to put a space after each comma in the function
parameter list.
Erasure Coding: Update safemode calculation for striped blocks
--
Key: HDFS-7864
URL: https://issues.apache.org/jira/browse/HDFS-7864
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: GAO Rui
Attachments: HDFS-7864.1.patch, HDFS-7864.2.patch
We need to update the safemode calculation for striped blocks. Specifically,
each striped block now consists of multiple data/parity blocks stored in
corresponding DataNodes. The current code's calculation is thus inconsistent:
each striped block is only counted as 1 expected block, while each of its
member block may increase the number of received blocks by 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7054) Make DFSOutputStream tracing more fine-grained
[ https://issues.apache.org/jira/browse/HDFS-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-7054: --- Attachment: HDFS-7054.002.patch Make DFSOutputStream tracing more fine-grained -- Key: HDFS-7054 URL: https://issues.apache.org/jira/browse/HDFS-7054 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 2.6.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7054.001.patch, HDFS-7054.002.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7933) fsck should also report decommissioning replicas.
[
https://issues.apache.org/jira/browse/HDFS-7933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364388#comment-14364388
]
Hadoop QA commented on HDFS-7933:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704903/HDFS-7933.00.patch
against trunk revision 2681ed9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestClientReportBadBlock
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9905//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9905//console
This message is automatically generated.
fsck should also report decommissioning replicas.
--
Key: HDFS-7933
URL: https://issues.apache.org/jira/browse/HDFS-7933
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Jitendra Nath Pandey
Assignee: Xiaoyu Yao
Attachments: HDFS-7933.00.patch
Fsck doesn't count replicas that are on decommissioning nodes. If a block has
all replicas on the decommissioning nodes, it will be marked as missing,
which is alarming for the admins, although the system will replicate them
before nodes are decommissioned.
Fsck output should also show decommissioning replicas along with the live
replicas.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7937) Erasure Coding: INodeFile quota computation unit tests
[ https://issues.apache.org/jira/browse/HDFS-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HDFS-7937: - Status: Patch Available (was: Open) [~jingzhao] I submitted an initial patch including clean up which is mentioned above. Can you review it? Thank you. Erasure Coding: INodeFile quota computation unit tests -- Key: HDFS-7937 URL: https://issues.apache.org/jira/browse/HDFS-7937 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Sasaki Assignee: Kai Sasaki Priority: Minor Attachments: HDFS-7937.1.patch Unit test for [HDFS-7826|https://issues.apache.org/jira/browse/HDFS-7826] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7937) Erasure Coding: INodeFile quota computation unit tests
[ https://issues.apache.org/jira/browse/HDFS-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HDFS-7937: - Attachment: HDFS-7937.1.patch Erasure Coding: INodeFile quota computation unit tests -- Key: HDFS-7937 URL: https://issues.apache.org/jira/browse/HDFS-7937 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Sasaki Assignee: Kai Sasaki Priority: Minor Attachments: HDFS-7937.1.patch Unit test for [HDFS-7826|https://issues.apache.org/jira/browse/HDFS-7826] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7054) Make DFSOutputStream tracing more fine-grained
[
https://issues.apache.org/jira/browse/HDFS-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364448#comment-14364448
]
Hadoop QA commented on HDFS-7054:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704958/HDFS-7054.002.patch
against trunk revision 5608520.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1155 javac
compiler warnings (more than the trunk's current 1152 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9909//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9909//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9909//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9909//console
This message is automatically generated.
Make DFSOutputStream tracing more fine-grained
--
Key: HDFS-7054
URL: https://issues.apache.org/jira/browse/HDFS-7054
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-7054.001.patch, HDFS-7054.002.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7616) Change FSImage to support BlockGroup
[ https://issues.apache.org/jira/browse/HDFS-7616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364496#comment-14364496 ] Takuya Fukudome commented on HDFS-7616: --- Hi [~szetszwo], I am interested in working on this task. Could you assign it to me? Thank you. Change FSImage to support BlockGroup Key: HDFS-7616 URL: https://issues.apache.org/jira/browse/HDFS-7616 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze We need to change FSImage for support BlockGroup and other new structures introduced for EC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7932) Speed up the shutdown of datanode during rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364236#comment-14364236
]
Hadoop QA commented on HDFS-7932:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704863/HDFS-7932.patch
against trunk revision ce5de93.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9904//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9904//console
This message is automatically generated.
Speed up the shutdown of datanode during rolling upgrade
Key: HDFS-7932
URL: https://issues.apache.org/jira/browse/HDFS-7932
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-7932.patch, HDFS-7932.patch
Datanode normally exits in 3 seconds after receiving {{shutdownDatanode}}
command. However, sometimes it doesn't, especially when the IO is busy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7054) Make DFSOutputStream tracing more fine-grained
[
https://issues.apache.org/jira/browse/HDFS-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364300#comment-14364300
]
Hadoop QA commented on HDFS-7054:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704936/HDFS-7054.001.patch
against trunk revision 5608520.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9907//console
This message is automatically generated.
Make DFSOutputStream tracing more fine-grained
--
Key: HDFS-7054
URL: https://issues.apache.org/jira/browse/HDFS-7054
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-7054.001.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7891) A block placement policy with best fault tolerance
[ https://issues.apache.org/jira/browse/HDFS-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7891: Summary: A block placement policy with best fault tolerance (was: a block placement policy with best fault tolerance) A block placement policy with best fault tolerance -- Key: HDFS-7891 URL: https://issues.apache.org/jira/browse/HDFS-7891 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Walter Su Assignee: Walter Su Attachments: HDFS-7891.002.patch, HDFS-7891.patch a block placement policy tries its best to place replicas to most racks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7912) Erasure Coding: track BlockInfo instead of Block in UnderReplicatedBlocks and PendingReplicationBlocks
[
https://issues.apache.org/jira/browse/HDFS-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364334#comment-14364334
]
Jing Zhao commented on HDFS-7912:
-
Thanks for the review, Zhe!
bq. My only concern is that it adds some getStoredBlock calls
The patch adds {{getStoredBlock}} call in mainly two places:
{{removeStoredBlock}} and {{addBlock}}. The {{addBlock}} function is currently
called to handle a RECEIVED_BLOCK msg in incremental block report, and it is
not happening when handling a full block report. Thus I guess here we will not
bring too much extra overhead to the NN.
bq. do we need storedBlock in the following code
Here my thought is that we should use BlockInfo for {{countNodes}}, since the
function tries to retrieve the storages from the blocksMap for the given block.
And with this we need to make the change in NameNodeFsck.
Erasure Coding: track BlockInfo instead of Block in UnderReplicatedBlocks and
PendingReplicationBlocks
--
Key: HDFS-7912
URL: https://issues.apache.org/jira/browse/HDFS-7912
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-7912.000.patch
Now with striped blocks and the design that uses a single BlockInfoStriped
object to track all the corresponding blocks, we need to clearly distinguish
the type Block and BlockInfo in BlockManager. Specifically, data structures
like {{UnderReplicatedBlocks}} and {{PendingReplicationBlocks}} should track
BlockInfo instead of Block in order to support striped block recovery.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7937) Erasure Coding: INodeFile quota computation unit tests
[ https://issues.apache.org/jira/browse/HDFS-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HDFS-7937: - Attachment: HDFS-7937.1.patch Erasure Coding: INodeFile quota computation unit tests -- Key: HDFS-7937 URL: https://issues.apache.org/jira/browse/HDFS-7937 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Sasaki Assignee: Kai Sasaki Priority: Minor Attachments: HDFS-7937.1.patch Unit test for [HDFS-7826|https://issues.apache.org/jira/browse/HDFS-7826] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7936) Erasure coding: resolving conflicts when merging with HDFS-7903 and HDFS-7435
[ https://issues.apache.org/jira/browse/HDFS-7936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364239#comment-14364239 ] Jing Zhao commented on HDFS-7936: - Yeah, but we do not have the class BlockInfo in trunk currently.. Erasure coding: resolving conflicts when merging with HDFS-7903 and HDFS-7435 - Key: HDFS-7936 URL: https://issues.apache.org/jira/browse/HDFS-7936 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Fix For: HDFS-7285 Attachments: HDFS-7936-001.patch, HDFS-7936-002.patch A few non-trivial conflicts were found when merging trunk changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7915) The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error
[
https://issues.apache.org/jira/browse/HDFS-7915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364392#comment-14364392
]
Colin Patrick McCabe commented on HDFS-7915:
Good catch. I committed to branch-2 but not branch-2.7, which was incorrect.
Fixed.
The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell
the DFSClient about it because of a network error
-
Key: HDFS-7915
URL: https://issues.apache.org/jira/browse/HDFS-7915
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Fix For: 2.7.0
Attachments: HDFS-7915.001.patch, HDFS-7915.002.patch,
HDFS-7915.004.patch, HDFS-7915.005.patch, HDFS-7915.006.patch
The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell
the DFSClient about it because of a network error. In
{{DataXceiver#requestShortCircuitFds}}, the DataNode can succeed at the first
part (mark the slot as used) and fail at the second part (tell the DFSClient
what it did). The try block for unregistering the slot only covers a
failure in the first part, not the second part. In this way, a divergence can
form between the views of which slots are allocated on DFSClient and on
server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7929) inotify unable fetch pre-upgrade edit log segments once upgrade starts
[
https://issues.apache.org/jira/browse/HDFS-7929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364411#comment-14364411
]
Colin Patrick McCabe commented on HDFS-7929:
The approach looks reasonable here.
{code}
131 File[] files = tmpDir.listFiles();
132 if (files != null) {
{code}
We should use {{IOUtils#listDirectory}} here. That function will throw an
exception if there is an IO error; in contrast, {{File#listFiles}} will
silently hide the error and just return null.
I am +1 pending that (and getting a good jenkins run)
We might want to file a follow-up JIRA for looking into whether we need any
changes in {{BackupJournalManager}} or {{BookKeeperJournalManager}}. Of
course, I think in most cases people configure local file journal managers to
go along with those journal managers, so it probably won't be an issue for most
systems.
inotify unable fetch pre-upgrade edit log segments once upgrade starts
--
Key: HDFS-7929
URL: https://issues.apache.org/jira/browse/HDFS-7929
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-7929-000.patch, HDFS-7929-001.patch
inotify is often used to periodically poll HDFS events. However, once an HDFS
upgrade has started, edit logs are moved to /previous on the NN, which is not
accessible. Moreover, once the upgrade is finalized /previous is currently
lost forever.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7838) Expose truncate API for libhdfs
[ https://issues.apache.org/jira/browse/HDFS-7838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364455#comment-14364455 ] Colin Patrick McCabe commented on HDFS-7838: +1. Thanks, Yi. Expose truncate API for libhdfs --- Key: HDFS-7838 URL: https://issues.apache.org/jira/browse/HDFS-7838 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 2.7.0 Reporter: Yi Liu Assignee: Yi Liu Attachments: HDFS-7838.001.patch, HDFS-7838.002.patch, HDFS-7838.003.patch, HDFS-7838.004.patch It's good to expose truncate in libhdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7837) Erasure Coding: allocate and persist striped blocks in NameNode
[ https://issues.apache.org/jira/browse/HDFS-7837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363837#comment-14363837 ] Zhe Zhang commented on HDFS-7837: - [~jingzhao] I rebased trunk changes today and manually fixed some conflicts. I also squashed the 2 HDFS-7837 commits. Please take a look at the [commit | https://git-wip-us.apache.org/repos/asf?p=hadoop.git;a=commitdiff;h=bc962947a19ed0a9ee130f62e819b65a64a8fb18] and see if it looks OK. Thanks! Erasure Coding: allocate and persist striped blocks in NameNode --- Key: HDFS-7837 URL: https://issues.apache.org/jira/browse/HDFS-7837 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Jing Zhao Fix For: HDFS-7285 Attachments: HDFS-7837.000.patch, HDFS-7837.001.patch Try to finish the remaining work from HDFS-7339 (except the ClientProtocol/DFSClient part): # Allow FSNamesystem#getAdditionalBlock to create striped blocks and persist striped blocks to editlog # Update FSImage for max allocated striped block ID # Update the block commit/complete logic in BlockManager -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7932) Speed up the shutdown of datanode during rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-7932:
-
Attachment: HDFS-7932.patch
Making it wait shorter for the dataXfer thread group to exit. (From 2.5 seconds
to 1 second)
Speed up the shutdown of datanode during rolling upgrade
Key: HDFS-7932
URL: https://issues.apache.org/jira/browse/HDFS-7932
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Attachments: HDFS-7932.patch, HDFS-7932.patch
Datanode normally exits in 3 seconds after receiving {{shutdownDatanode}}
command. However, sometimes it doesn't, especially when the IO is busy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7935) Support multi-homed networks when Kerberos security is enabled
[ https://issues.apache.org/jira/browse/HDFS-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363175#comment-14363175 ] Kihwal Lee commented on HDFS-7935: -- Have you tried HADOOP-9789? For example, you can configure dfs.namenode.kerberos.principal.pattern on the client side to make it accept a SPN that is different from the connecting namenode address. Support multi-homed networks when Kerberos security is enabled -- Key: HDFS-7935 URL: https://issues.apache.org/jira/browse/HDFS-7935 Project: Hadoop HDFS Issue Type: Bug Reporter: Arun Suresh Assignee: Arun Suresh Currently, during SASL negotiation stage between ipc Client and Server, The server sends only a single serviceId (curresponding to a single principal) to the client. This is the principal the the server process is logged in as during startup. It is possible that in a multi-homed network, the server might be associated with more than one principal, and thus severs must provide the clients all possible principals it can use to connect to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7060) Contentions of the monitor of FsDatasetImpl block DN's heartbeat
[
https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363215#comment-14363215
]
Hadoop QA commented on HDFS-7060:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704754/HDFS-7060.001.patch
against trunk revision 3ff1ba2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9896//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9896//console
This message is automatically generated.
Contentions of the monitor of FsDatasetImpl block DN's heartbeat
Key: HDFS-7060
URL: https://issues.apache.org/jira/browse/HDFS-7060
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Xinwei Qin
Attachments: HDFS-7060.000.patch, HDFS-7060.001.patch
We're seeing the heartbeat is blocked by the monitor of {{FsDatasetImpl}}
when the DN is under heavy load of writes:
{noformat}
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getDfsUsed(FsVolumeImpl.java:115)
- waiting to lock 0x000780304fb8 (a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getStorageReports(FsDatasetImpl.java:91)
- locked 0x000780612fd8 (a java.lang.Object)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:563)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:668)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:827)
at java.lang.Thread.run(Thread.java:744)
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:743)
- waiting to lock 0x000780304fb8 (a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:60)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:169)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:621)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:232)
at java.lang.Thread.run(Thread.java:744)
java.lang.Thread.State: RUNNABLE
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1006)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:59)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:244)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:195)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:753)
- locked 0x000780304fb8 (a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
[jira] [Commented] (HDFS-7854) Separate class DataStreamer out of DFSOutputStream
[
https://issues.apache.org/jira/browse/HDFS-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363169#comment-14363169
]
Hadoop QA commented on HDFS-7854:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704748/HDFS-7854-004-duplicate2.patch
against trunk revision 3ff1ba2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9895//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9895//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9895//console
This message is automatically generated.
Separate class DataStreamer out of DFSOutputStream
--
Key: HDFS-7854
URL: https://issues.apache.org/jira/browse/HDFS-7854
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Li Bo
Assignee: Li Bo
Attachments: HDFS-7854-001.patch, HDFS-7854-002.patch,
HDFS-7854-003.patch, HDFS-7854-004-duplicate.patch,
HDFS-7854-004-duplicate2.patch, HDFS-7854-004.patch
This sub task separate DataStreamer from DFSOutputStream. New DataStreamer
will accept packets and write them to remote datanodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[
https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated HDFS-7847:
---
Attachment: HDFS-7847.002.patch
@cmccabe, @stack, thanks for the review!
bq. DFSClient.java: this change adds three new fields to DFSClient. But they
only seem to be used by unit tests. It seems like we should just put these
inside the unit test(s) that are using these-- if necessary, by adding a helper
method. There's no reason to add more fields to DFSClient. Also remember that
when using FileContext, we create new DFSClients all the time.
Good point. I've left the existing {code}ClientProtocol namenode{code} field
alone. The other 3 proxies are created on-demand by their getters. That means
no change in DFSClient instance size.
bq. It seems kind of odd to have NameNodeProxies#createProxy create a proxy to
the datanode.
It's actually a proxy to the NN for the DatanodeProtocol. That's the same
protocol that the DN uses to speak with the NN when it's sending (among other
things) block reports. But with some ideas from @stack, I got rid of the
changes to NameNodeProxies.
bq. Of course the NameNode may or may not be remote here. It seems like --nnuri
or just --namenode or something like that would be more descriptive.
Yeah, I agree. I changed it to -namenode.
bq. Instead of this boilerplate, just use StringUtils#popOptionWithArgument.
Changed. I was just trying to match the existing code, but the using
StringUtils is for the better.
{code}
- replication, BLOCK_SIZE, null);
+ replication, BLOCK_SIZE, CryptoProtocolVersion.supported());
{code}
bq. This fix is a little bit separate, right? I suppose we can do it in this
JIRA, though.
Without this, the relevant PBHelper.convert code throws NPE on the
supportVersions arg.
Modify NNThroughputBenchmark to be able to operate on a remote NameNode
---
Key: HDFS-7847
URL: https://issues.apache.org/jira/browse/HDFS-7847
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7836
Reporter: Colin Patrick McCabe
Assignee: Charles Lamb
Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch,
HDFS-7847.002.patch, make_blocks.tar.gz
Modify NNThroughputBenchmark to be able to operate on a NN that is not in
process. A followon Jira will modify it some more to allow quantifying native
and java heap sizes, and some latency numbers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5356) MiniDFSCluster shoud close all open FileSystems when shutdown()
[
https://issues.apache.org/jira/browse/HDFS-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363045#comment-14363045
]
Hadoop QA commented on HDFS-5356:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704667/HDFS-5356-8.patch
against trunk revision 3ff1ba2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9892//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9892//console
This message is automatically generated.
MiniDFSCluster shoud close all open FileSystems when shutdown()
---
Key: HDFS-5356
URL: https://issues.apache.org/jira/browse/HDFS-5356
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.2.0
Reporter: haosdent
Assignee: Rakesh R
Priority: Critical
Attachments: HDFS-5356-1.patch, HDFS-5356-2.patch, HDFS-5356-3.patch,
HDFS-5356-4.patch, HDFS-5356-5.patch, HDFS-5356-6.patch, HDFS-5356-7.patch,
HDFS-5356-8.patch, HDFS-5356.patch
After add some metrics functions to DFSClient, I found that some unit tests
relates to metrics are failed. Because MiniDFSCluster are never close open
FileSystems, DFSClients are alive after MiniDFSCluster shutdown(). The
metrics of DFSClients in DefaultMetricsSystem are still exist and this make
other unit tests failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-3325) When configuring dfs.namenode.safemode.threshold-pct to a value greater or equal to 1 there is mismatch in the UI report
[
https://issues.apache.org/jira/browse/HDFS-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-3325:
-
Attachment: HDFS-3325.2.patch
Hi Vinayakumar B ,
Attached a patch with updated testcases.
Please review.
When configuring dfs.namenode.safemode.threshold-pct to a value greater or
equal to 1 there is mismatch in the UI report
--
Key: HDFS-3325
URL: https://issues.apache.org/jira/browse/HDFS-3325
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: J.Andreina
Assignee: J.Andreina
Priority: Minor
Attachments: HDFS-3325.1.patch, HDFS-3325.2.patch
When dfs.namenode.safemode.threshold-pct is configured to n
Namenode will be in safemode until n percentage of blocks that should satisfy
the minimal replication requirement defined by
dfs.namenode.replication.min is reported to namenode
But in UI it displays that n percentage of total blocks + 1 blocks are
additionally needed
to come out of the safemode
Scenario 1:
Configurations:
dfs.namenode.safemode.threshold-pct = 2
dfs.replication = 2
dfs.namenode.replication.min =2
Step 1: Start NN,DN1,DN2
Step 2: Write a file a.txt which has got 167 blocks
step 3: Stop NN,DN1,DN2
Step 4: start NN
In UI report the Number of blocks needed to come out of safemode and number
of blocks actually present is different.
{noformat}
Cluster Summary
Security is OFF
Safe mode is ON. The reported blocks 0 needs additional 335 blocks to reach
the threshold 2. of total blocks 167. Safe mode will be turned off
automatically.
2 files and directories, 167 blocks = 169 total.
Heap Memory used 57.05 MB is 2% of Commited Heap Memory 2 GB. Max Heap Memory
is 2 GB.
Non Heap Memory used 23.37 MB is 17% of Commited Non Heap Memory 130.44 MB.
Max Non Heap Memory is 176 MB.{noformat}
Scenario 2:
===
Configurations:
dfs.namenode.safemode.threshold-pct = 1
dfs.replication = 2
dfs.namenode.replication.min =2
Step 1: Start NN,DN1,DN2
Step 2: Write a file a.txt which has got 167 blocks
step 3: Stop NN,DN1,DN2
Step 4: start NN
In UI report the Number of blocks needed to come out of safemode and number
of blocks actually present is different
{noformat}
Cluster Summary
Security is OFF
Safe mode is ON. The reported blocks 0 needs additional 168 blocks to reach
the threshold 1. of total blocks 167. Safe mode will be turned off
automatically.
2 files and directories, 167 blocks = 169 total.
Heap Memory used 56.2 MB is 2% of Commited Heap Memory 2 GB. Max Heap Memory
is 2 GB.
Non Heap Memory used 23.37 MB is 17% of Commited Non Heap Memory 130.44 MB.
Max Non Heap Memory is 176 MB.{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7838) Expose truncate API for libhdfs
[
https://issues.apache.org/jira/browse/HDFS-7838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363126#comment-14363126
]
Hadoop QA commented on HDFS-7838:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704742/HDFS-7838.004.patch
against trunk revision 3ff1ba2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager
org.apache.hadoop.hdfs.server.balancer.TestBalancer
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestNameNodeXAttr
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9894//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9894//console
This message is automatically generated.
Expose truncate API for libhdfs
---
Key: HDFS-7838
URL: https://issues.apache.org/jira/browse/HDFS-7838
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 2.7.0
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-7838.001.patch, HDFS-7838.002.patch,
HDFS-7838.003.patch, HDFS-7838.004.patch
It's good to expose truncate in libhdfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-3325) When configuring dfs.namenode.safemode.threshold-pct to a value greater or equal to 1 there is mismatch in the UI report
[
https://issues.apache.org/jira/browse/HDFS-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363065#comment-14363065
]
Hadoop QA commented on HDFS-3325:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704758/HDFS-3325.2.patch
against trunk revision 3ff1ba2.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9897//console
This message is automatically generated.
When configuring dfs.namenode.safemode.threshold-pct to a value greater or
equal to 1 there is mismatch in the UI report
--
Key: HDFS-3325
URL: https://issues.apache.org/jira/browse/HDFS-3325
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: J.Andreina
Assignee: J.Andreina
Priority: Minor
Attachments: HDFS-3325.1.patch, HDFS-3325.2.patch
When dfs.namenode.safemode.threshold-pct is configured to n
Namenode will be in safemode until n percentage of blocks that should satisfy
the minimal replication requirement defined by
dfs.namenode.replication.min is reported to namenode
But in UI it displays that n percentage of total blocks + 1 blocks are
additionally needed
to come out of the safemode
Scenario 1:
Configurations:
dfs.namenode.safemode.threshold-pct = 2
dfs.replication = 2
dfs.namenode.replication.min =2
Step 1: Start NN,DN1,DN2
Step 2: Write a file a.txt which has got 167 blocks
step 3: Stop NN,DN1,DN2
Step 4: start NN
In UI report the Number of blocks needed to come out of safemode and number
of blocks actually present is different.
{noformat}
Cluster Summary
Security is OFF
Safe mode is ON. The reported blocks 0 needs additional 335 blocks to reach
the threshold 2. of total blocks 167. Safe mode will be turned off
automatically.
2 files and directories, 167 blocks = 169 total.
Heap Memory used 57.05 MB is 2% of Commited Heap Memory 2 GB. Max Heap Memory
is 2 GB.
Non Heap Memory used 23.37 MB is 17% of Commited Non Heap Memory 130.44 MB.
Max Non Heap Memory is 176 MB.{noformat}
Scenario 2:
===
Configurations:
dfs.namenode.safemode.threshold-pct = 1
dfs.replication = 2
dfs.namenode.replication.min =2
Step 1: Start NN,DN1,DN2
Step 2: Write a file a.txt which has got 167 blocks
step 3: Stop NN,DN1,DN2
Step 4: start NN
In UI report the Number of blocks needed to come out of safemode and number
of blocks actually present is different
{noformat}
Cluster Summary
Security is OFF
Safe mode is ON. The reported blocks 0 needs additional 168 blocks to reach
the threshold 1. of total blocks 167. Safe mode will be turned off
automatically.
2 files and directories, 167 blocks = 169 total.
Heap Memory used 56.2 MB is 2% of Commited Heap Memory 2 GB. Max Heap Memory
is 2 GB.
Non Heap Memory used 23.37 MB is 17% of Commited Non Heap Memory 130.44 MB.
Max Non Heap Memory is 176 MB.{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7060) Contentions of the monitor of FsDatasetImpl block DN's heartbeat
[
https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinwei Qin updated HDFS-7060:
--
Attachment: HDFS-7060.001.patch
Contentions of the monitor of FsDatasetImpl block DN's heartbeat
Key: HDFS-7060
URL: https://issues.apache.org/jira/browse/HDFS-7060
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Xinwei Qin
Attachments: HDFS-7060.000.patch, HDFS-7060.001.patch
We're seeing the heartbeat is blocked by the monitor of {{FsDatasetImpl}}
when the DN is under heavy load of writes:
{noformat}
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getDfsUsed(FsVolumeImpl.java:115)
- waiting to lock 0x000780304fb8 (a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getStorageReports(FsDatasetImpl.java:91)
- locked 0x000780612fd8 (a java.lang.Object)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:563)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:668)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:827)
at java.lang.Thread.run(Thread.java:744)
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:743)
- waiting to lock 0x000780304fb8 (a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:60)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:169)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:621)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:232)
at java.lang.Thread.run(Thread.java:744)
java.lang.Thread.State: RUNNABLE
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1006)
at
org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:59)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:244)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:195)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:753)
- locked 0x000780304fb8 (a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:60)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.init(BlockReceiver.java:169)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:621)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:232)
at java.lang.Thread.run(Thread.java:744)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7267) TestBalancer#testUnknownDatanode occasionally fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363070#comment-14363070
]
Walter Su commented on HDFS-7267:
-
set block report to 1s is not a decent solution. I can't think up a better
solution.
TestBalancer#testUnknownDatanode occasionally fails in trunk
Key: HDFS-7267
URL: https://issues.apache.org/jira/browse/HDFS-7267
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
Attachments: testUnknownDatanode-failed-log.html
In build #1907 (https://builds.apache.org/job/Hadoop-Hdfs-trunk/1907/):
{code}
REGRESSION:
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testUnknownDatanode
Error Message:
expected:0 but was:-3
Stack Trace:
java.lang.AssertionError: expected:0 but was:-3
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testUnknownDatanode(TestBalancer.java:737)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7934) During Rolling upgrade rollback ,standby namenode startup fails.
[
https://issues.apache.org/jira/browse/HDFS-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14362978#comment-14362978
]
Vinayakumar B commented on HDFS-7934:
-
I think the problematic area is below code block in Fsimage#loadFsImage(..)
{code} // For rollback in rolling upgrade, we need to set the
toAtLeastTxId to
// the txid right before the upgrade marker.
long toAtLeastTxId = editLog.isOpenForWrite() ? inspector
.getMaxSeenTxId() : 0;
if (rollingRollback) {
// note that the first image in imageFiles is the special checkpoint
// for the rolling upgrade
toAtLeastTxId = imageFiles.get(0).getCheckpointTxId() + 2;
}{code}
In Case of rollingRollback, there is nothing read from edits streams. So
setting {{toAtLeastTxId = imageFiles.get(0).getCheckpointTxId() + 2;}} is not
required. removing this line will solve the problem IMO.
Any thoughts?
During Rolling upgrade rollback ,standby namenode startup fails.
Key: HDFS-7934
URL: https://issues.apache.org/jira/browse/HDFS-7934
Project: Hadoop HDFS
Issue Type: Bug
Reporter: J.Andreina
Assignee: J.Andreina
Priority: Critical
During Rolling upgrade rollback , standby namenode startup fails , while
loading edits and when there is no local copy of edits created after upgrade
( which is already been removed by Active Namenode from journal manager and
from Active's local).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7930) commitBlockSynchronization() does not remove locations
[
https://issues.apache.org/jira/browse/HDFS-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-7930:
-
Status: Patch Available (was: Open)
commitBlockSynchronization() does not remove locations
--
Key: HDFS-7930
URL: https://issues.apache.org/jira/browse/HDFS-7930
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.7.0
Reporter: Konstantin Shvachko
Assignee: Yi Liu
Priority: Blocker
Attachments: HDFS-7930.001.patch
When {{commitBlockSynchronization()}} has less {{newTargets}} than in the
original block it does not remove unconfirmed locations. This results in that
the the block stores locations of different lengths or genStamp (corrupt).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-3325) When configuring dfs.namenode.safemode.threshold-pct to a value greater or equal to 1 there is mismatch in the UI report
[
https://issues.apache.org/jira/browse/HDFS-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363069#comment-14363069
]
J.Andreina commented on HDFS-3325:
--
Failures are not related to this patch
When configuring dfs.namenode.safemode.threshold-pct to a value greater or
equal to 1 there is mismatch in the UI report
--
Key: HDFS-3325
URL: https://issues.apache.org/jira/browse/HDFS-3325
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: J.Andreina
Assignee: J.Andreina
Priority: Minor
Attachments: HDFS-3325.1.patch, HDFS-3325.2.patch
When dfs.namenode.safemode.threshold-pct is configured to n
Namenode will be in safemode until n percentage of blocks that should satisfy
the minimal replication requirement defined by
dfs.namenode.replication.min is reported to namenode
But in UI it displays that n percentage of total blocks + 1 blocks are
additionally needed
to come out of the safemode
Scenario 1:
Configurations:
dfs.namenode.safemode.threshold-pct = 2
dfs.replication = 2
dfs.namenode.replication.min =2
Step 1: Start NN,DN1,DN2
Step 2: Write a file a.txt which has got 167 blocks
step 3: Stop NN,DN1,DN2
Step 4: start NN
In UI report the Number of blocks needed to come out of safemode and number
of blocks actually present is different.
{noformat}
Cluster Summary
Security is OFF
Safe mode is ON. The reported blocks 0 needs additional 335 blocks to reach
the threshold 2. of total blocks 167. Safe mode will be turned off
automatically.
2 files and directories, 167 blocks = 169 total.
Heap Memory used 57.05 MB is 2% of Commited Heap Memory 2 GB. Max Heap Memory
is 2 GB.
Non Heap Memory used 23.37 MB is 17% of Commited Non Heap Memory 130.44 MB.
Max Non Heap Memory is 176 MB.{noformat}
Scenario 2:
===
Configurations:
dfs.namenode.safemode.threshold-pct = 1
dfs.replication = 2
dfs.namenode.replication.min =2
Step 1: Start NN,DN1,DN2
Step 2: Write a file a.txt which has got 167 blocks
step 3: Stop NN,DN1,DN2
Step 4: start NN
In UI report the Number of blocks needed to come out of safemode and number
of blocks actually present is different
{noformat}
Cluster Summary
Security is OFF
Safe mode is ON. The reported blocks 0 needs additional 168 blocks to reach
the threshold 1. of total blocks 167. Safe mode will be turned off
automatically.
2 files and directories, 167 blocks = 169 total.
Heap Memory used 56.2 MB is 2% of Commited Heap Memory 2 GB. Max Heap Memory
is 2 GB.
Non Heap Memory used 23.37 MB is 17% of Commited Non Heap Memory 130.44 MB.
Max Non Heap Memory is 176 MB.{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-3745) fsck prints that it's using KSSL even when it's in fact using SPNEGO for authentication
[
https://issues.apache.org/jira/browse/HDFS-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HDFS-3745:
--
Target Version/s: (was: 2.0.2-alpha)
fsck prints that it's using KSSL even when it's in fact using SPNEGO for
authentication
---
Key: HDFS-3745
URL: https://issues.apache.org/jira/browse/HDFS-3745
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client, security
Affects Versions: 1.2.0, 2.0.0-alpha
Reporter: Aaron T. Myers
Priority: Trivial
Labels: newbie
In branch-2 (which exclusively uses SPNEGO for HTTP authentication) and in
branch-1 (which can optionally use SPNEGO for HTTP authentication), running
fsck will print the following, which isn't quite right:
{quote}
FSCK started by hdfs (auth:KERBEROS_SSL) from...
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-3745) fsck prints that it's using KSSL even when it's in fact using SPNEGO for authentication
[
https://issues.apache.org/jira/browse/HDFS-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HDFS-3745:
--
Affects Version/s: (was: 1.2.0)
fsck prints that it's using KSSL even when it's in fact using SPNEGO for
authentication
---
Key: HDFS-3745
URL: https://issues.apache.org/jira/browse/HDFS-3745
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client, security
Affects Versions: 2.0.0-alpha
Reporter: Aaron T. Myers
Priority: Trivial
Labels: newbie
In branch-2 (which exclusively uses SPNEGO for HTTP authentication) and in
branch-1 (which can optionally use SPNEGO for HTTP authentication), running
fsck will print the following, which isn't quite right:
{quote}
FSCK started by hdfs (auth:KERBEROS_SSL) from...
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-4829) Make the fs-shell tail command behave more like Unix tail command
[ https://issues.apache.org/jira/browse/HDFS-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-4829: -- Labels: newbie (was: ) Make the fs-shell tail command behave more like Unix tail command - Key: HDFS-4829 URL: https://issues.apache.org/jira/browse/HDFS-4829 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.0.0-alpha Reporter: Todd Grayson Priority: Minor Labels: newbie Strange behavior of the hadoop fs -tail command - its default for output seems to be 9 lines of output vs 10 lines of output in the OS version of the command (minor issue). The strange thing (bug behavior?) appears to drop the initial octect from an IP address when examining a file over HDFS. [training@localhost hands-on]$ hadoop fs -tail weblog/access_log .190.174.142 - - [03/Dec/2011:13:28:08 -0800] GET /assets/js/javascript_combined.js HTTP/1.1 200 20404 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /assets/img/home-logo.png HTTP/1.1 200 3892 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/019.jpg HTTP/1.1 200 74446 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/g_still_04.jpg HTTP/1.1 200 761555 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/07082218.jpg HTTP/1.1 200 154609 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000163.jpg HTTP/1.1 200 60117 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/Chacha.jpg HTTP/1.1 200 109379 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000159.jpg HTTP/1.1 200 161657 *When looking at the original log data outside of HDFS with the os version of the tail command we see the following* [training@localhost hands-on]$ hadoop fs -get weblog/access_log ./ [training@localhost hands-on]$ tail access_log 10.190.174.142 - - [03/Dec/2011:13:28:06 -0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 10.190.174.142 - - [03/Dec/2011:13:28:08 -0800] GET /assets/js/javascript_combined.js HTTP/1.1 200 20404 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /assets/img/home-logo.png HTTP/1.1 200 3892 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/019.jpg HTTP/1.1 200 74446 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/g_still_04.jpg HTTP/1.1 200 761555 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/07082218.jpg HTTP/1.1 200 154609 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000163.jpg HTTP/1.1 200 60117 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/Chacha.jpg HTTP/1.1 200 109379 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000159.jpg HTTP/1.1 200 161657 When using non ip data seperated by periods, it gets even worse and even more data is masked? (same data subtituting names for IP octects). Note we loose the first line well into the URI string? * [training@localhost hands-on]$ hadoop fs -tail weblog/test_log s/javascript_combined.js HTTP/1.1 200 20404 larry.billy.will.amy - - [03/Dec/2011:13:28:09 -0800] GET /assets/img/home-logo.png HTTP/1.1 200 3892 larry.billy.will.amy - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/019.jpg HTTP/1.1 200 74446 larry.billy.will.amy - - [03/Dec/2011:13:28:larry.-0800] GET /images/filmmediablock/360/g_still_04.jpg HTTP/1.1 200 761555 larry.billy.will.amy - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/07082218.jpg HTTP/1.1 200 154609 larry.billy.will.amy - - [03/Dec/2011:13:28:larry.-0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 larry.billy.will.amy - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000163.jpg HTTP/1.1 200 60117 larry.billy.will.amy - - [03/Dec/2011:13:28:larry.-0800] GET /images/filmmediablock/360/Chacha.jpg HTTP/1.1 200 larry.379 larry.billy.will.amy - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000159.jpg HTTP/1.1 200 161657 * and verifying what we are looking at in normal tail matches - note the first line is not represented in the hadoop fs -tail as its only grabbing 9 lines instead of 10... as I mentioned before. Align the two text
[jira] [Commented] (HDFS-7935) Support multi-homed networks when Kerberos security is enabled
[ https://issues.apache.org/jira/browse/HDFS-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363477#comment-14363477 ] Arun Suresh commented on HDFS-7935: --- [~kihwal], Yup looks like that would work. I guess then.. that the only thing remaining is a minor fix for https://issues.apache.org/jira/browse/HDFS-4448 which would allow one to deploy NN in HA with security in a multi-homed env. Support multi-homed networks when Kerberos security is enabled -- Key: HDFS-7935 URL: https://issues.apache.org/jira/browse/HDFS-7935 Project: Hadoop HDFS Issue Type: Bug Reporter: Arun Suresh Assignee: Arun Suresh Currently, during SASL negotiation stage between ipc Client and Server, The server sends only a single serviceId (curresponding to a single principal) to the client. This is the principal the the server process is logged in as during startup. It is possible that in a multi-homed network, the server might be associated with more than one principal, and thus severs must provide the clients all possible principals it can use to connect to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-2360) Ugly stacktrace when quota exceeds
[
https://issues.apache.org/jira/browse/HDFS-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved HDFS-2360.
---
Resolution: Not a Problem
The last line of the command (excluding the log and its stack trace via the
WARN) does today print the base message reason that should catch the eye
clearly:
{code}
put: The DiskSpace quota of /testDir is exceeded: quota = 1024 B = 1 KB but
diskspace consumed = 402653184 B = 384 MB
{code}
Resolving this as it should be clear enough. To get rid of the WARN, the client
logger can be nullified, but the catch layer is rather generic today to
specifically turn it off without causing other impact (for other use-cases and
troubles) I think.
As always though, feel free to reopen with any counter-point.
Ugly stacktrace when quota exceeds
--
Key: HDFS-2360
URL: https://issues.apache.org/jira/browse/HDFS-2360
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 0.23.0
Reporter: Rajit Saha
Priority: Minor
Will it be better to catch the exception and throw a small reasonable messege
to user when they exceed quota?
$hdfs dfs -mkdir testDir
$hdfs dfsadmin -setSpaceQuota 191M testDir
$hdfs dfs -count -q testDir
none inf 200278016 2002780161
0 0
hdfs://NN hostname:port/user/hdfsqa/testDir
$hdfs dfs -put /etc/passwd /user/hadoopqa/testDir
11/09/19 08:08:15 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota
of /user/hdfsqa/testDir is exceeded:
quota=191.0m diskspace consumed=768.0m
at
org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:159)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1609)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1383)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:370)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1681)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1476)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:389)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1496)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1492)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1490)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1100)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:972)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454)
Caused by: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The
DiskSpace quota of /user/hdfsqa/testDir is
exceeded: quota=191.0m diskspace consumed=768.0m
at
org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:159)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1609)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1383)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:370)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1681)
at
[jira] [Commented] (HDFS-5356) MiniDFSCluster shoud close all open FileSystems when shutdown()
[ https://issues.apache.org/jira/browse/HDFS-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363428#comment-14363428 ] Rakesh R commented on HDFS-5356: Thanks [~vinayrpet] for triggering the build. It looks like the testcase failure is unrelated to this patch. Please have a look at the latest patch, I addressed the review comments. MiniDFSCluster shoud close all open FileSystems when shutdown() --- Key: HDFS-5356 URL: https://issues.apache.org/jira/browse/HDFS-5356 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 3.0.0, 2.2.0 Reporter: haosdent Assignee: Rakesh R Priority: Critical Attachments: HDFS-5356-1.patch, HDFS-5356-2.patch, HDFS-5356-3.patch, HDFS-5356-4.patch, HDFS-5356-5.patch, HDFS-5356-6.patch, HDFS-5356-7.patch, HDFS-5356-8.patch, HDFS-5356.patch After add some metrics functions to DFSClient, I found that some unit tests relates to metrics are failed. Because MiniDFSCluster are never close open FileSystems, DFSClients are alive after MiniDFSCluster shutdown(). The metrics of DFSClients in DefaultMetricsSystem are still exist and this make other unit tests failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5640) Add snapshot methods to FileContext.
[
https://issues.apache.org/jira/browse/HDFS-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363432#comment-14363432
]
Hadoop QA commented on HDFS-5640:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704411/HDFS-5640-001.patch
against trunk revision d1eebd9.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9899//console
This message is automatically generated.
Add snapshot methods to FileContext.
Key: HDFS-5640
URL: https://issues.apache.org/jira/browse/HDFS-5640
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client, snapshots
Affects Versions: 3.0.0, 2.2.0
Reporter: Chris Nauroth
Assignee: Rakesh R
Attachments: HDFS-5640-001.patch
Currently, methods related to HDFS snapshots are defined on {{FileSystem}}.
For feature parity, these methods need to be added to {{FileContext}}. This
would also require updating {{AbstractFileSystem}} and the {{Hdfs}} subclass.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-4829) Make the fs-shell tail command behave more like Unix tail command
[ https://issues.apache.org/jira/browse/HDFS-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-4829: -- Summary: Make the fs-shell tail command behave more like Unix tail command (was: Strange loss of data displayed in hadoop fs -tail command) Make the fs-shell tail command behave more like Unix tail command - Key: HDFS-4829 URL: https://issues.apache.org/jira/browse/HDFS-4829 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.0.0-alpha Environment: OS Centos 6.3 (on Intel Core2 Duo, VMware Player VM running under windows 7) Testing on both 2.0.0-cdh4.1.1 and 2.0.0-cdh4.1.2 Reporter: Todd Grayson Priority: Minor Strange behavior of the hadoop fs -tail command - its default for output seems to be 9 lines of output vs 10 lines of output in the OS version of the command (minor issue). The strange thing (bug behavior?) appears to drop the initial octect from an IP address when examining a file over HDFS. [training@localhost hands-on]$ hadoop fs -tail weblog/access_log .190.174.142 - - [03/Dec/2011:13:28:08 -0800] GET /assets/js/javascript_combined.js HTTP/1.1 200 20404 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /assets/img/home-logo.png HTTP/1.1 200 3892 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/019.jpg HTTP/1.1 200 74446 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/g_still_04.jpg HTTP/1.1 200 761555 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/07082218.jpg HTTP/1.1 200 154609 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000163.jpg HTTP/1.1 200 60117 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/Chacha.jpg HTTP/1.1 200 109379 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000159.jpg HTTP/1.1 200 161657 *When looking at the original log data outside of HDFS with the os version of the tail command we see the following* [training@localhost hands-on]$ hadoop fs -get weblog/access_log ./ [training@localhost hands-on]$ tail access_log 10.190.174.142 - - [03/Dec/2011:13:28:06 -0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 10.190.174.142 - - [03/Dec/2011:13:28:08 -0800] GET /assets/js/javascript_combined.js HTTP/1.1 200 20404 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /assets/img/home-logo.png HTTP/1.1 200 3892 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/019.jpg HTTP/1.1 200 74446 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/g_still_04.jpg HTTP/1.1 200 761555 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/07082218.jpg HTTP/1.1 200 154609 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000163.jpg HTTP/1.1 200 60117 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/Chacha.jpg HTTP/1.1 200 109379 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000159.jpg HTTP/1.1 200 161657 When using non ip data seperated by periods, it gets even worse and even more data is masked? (same data subtituting names for IP octects). Note we loose the first line well into the URI string? * [training@localhost hands-on]$ hadoop fs -tail weblog/test_log s/javascript_combined.js HTTP/1.1 200 20404 larry.billy.will.amy - - [03/Dec/2011:13:28:09 -0800] GET /assets/img/home-logo.png HTTP/1.1 200 3892 larry.billy.will.amy - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/019.jpg HTTP/1.1 200 74446 larry.billy.will.amy - - [03/Dec/2011:13:28:larry.-0800] GET /images/filmmediablock/360/g_still_04.jpg HTTP/1.1 200 761555 larry.billy.will.amy - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/07082218.jpg HTTP/1.1 200 154609 larry.billy.will.amy - - [03/Dec/2011:13:28:larry.-0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 larry.billy.will.amy - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000163.jpg HTTP/1.1 200 60117 larry.billy.will.amy - - [03/Dec/2011:13:28:larry.-0800] GET /images/filmmediablock/360/Chacha.jpg HTTP/1.1 200 larry.379 larry.billy.will.amy - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000159.jpg
[jira] [Commented] (HDFS-3745) fsck prints that it's using KSSL even when it's in fact using SPNEGO for authentication
[
https://issues.apache.org/jira/browse/HDFS-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363409#comment-14363409
]
Harsh J commented on HDFS-3745:
---
I think it should not be harmful to keep the sub-mechanism info around. We
could just rename the KERBEROS_SSL into KERBEROS_SPNEGO on trunk I think, since
we do not use/support KERBEROS_SSL (KSSL) anymore in trunk and branch-2?
fsck prints that it's using KSSL even when it's in fact using SPNEGO for
authentication
---
Key: HDFS-3745
URL: https://issues.apache.org/jira/browse/HDFS-3745
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client, security
Affects Versions: 1.2.0, 2.0.0-alpha
Reporter: Aaron T. Myers
Priority: Trivial
Labels: newbie
In branch-2 (which exclusively uses SPNEGO for HTTP authentication) and in
branch-1 (which can optionally use SPNEGO for HTTP authentication), running
fsck will print the following, which isn't quite right:
{quote}
FSCK started by hdfs (auth:KERBEROS_SSL) from...
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7930) commitBlockSynchronization() does not remove locations
[
https://issues.apache.org/jira/browse/HDFS-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363443#comment-14363443
]
Hadoop QA commented on HDFS-7930:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704734/HDFS-7930.001.patch
against trunk revision 3ff1ba2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9898//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9898//console
This message is automatically generated.
commitBlockSynchronization() does not remove locations
--
Key: HDFS-7930
URL: https://issues.apache.org/jira/browse/HDFS-7930
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.7.0
Reporter: Konstantin Shvachko
Assignee: Yi Liu
Priority: Blocker
Attachments: HDFS-7930.001.patch
When {{commitBlockSynchronization()}} has less {{newTargets}} than in the
original block it does not remove unconfirmed locations. This results in that
the the block stores locations of different lengths or genStamp (corrupt).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-3349) DFSAdmin fetchImage command should initialize security credentials
[ https://issues.apache.org/jira/browse/HDFS-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HDFS-3349. --- Resolution: Cannot Reproduce Target Version/s: (was: 2.0.0-alpha) Trying with lack of credentials throws the proper response back (No tgt). I think this is stale given Aaron's comment as well, marking as resolved. DFSAdmin fetchImage command should initialize security credentials -- Key: HDFS-3349 URL: https://issues.apache.org/jira/browse/HDFS-3349 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Affects Versions: 2.0.0-alpha Reporter: Aaron T. Myers Priority: Minor The `hdfs dfsadmin -fetchImage' command should fetch the fsimage using the appropriate credentials if security is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-4829) Make the fs-shell tail command behave more like Unix tail command
[ https://issues.apache.org/jira/browse/HDFS-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-4829: -- Environment: (was: OS Centos 6.3 (on Intel Core2 Duo, VMware Player VM running under windows 7) Testing on both 2.0.0-cdh4.1.1 and 2.0.0-cdh4.1.2) Make the fs-shell tail command behave more like Unix tail command - Key: HDFS-4829 URL: https://issues.apache.org/jira/browse/HDFS-4829 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.0.0-alpha Reporter: Todd Grayson Priority: Minor Labels: newbie Strange behavior of the hadoop fs -tail command - its default for output seems to be 9 lines of output vs 10 lines of output in the OS version of the command (minor issue). The strange thing (bug behavior?) appears to drop the initial octect from an IP address when examining a file over HDFS. [training@localhost hands-on]$ hadoop fs -tail weblog/access_log .190.174.142 - - [03/Dec/2011:13:28:08 -0800] GET /assets/js/javascript_combined.js HTTP/1.1 200 20404 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /assets/img/home-logo.png HTTP/1.1 200 3892 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/019.jpg HTTP/1.1 200 74446 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/g_still_04.jpg HTTP/1.1 200 761555 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/07082218.jpg HTTP/1.1 200 154609 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000163.jpg HTTP/1.1 200 60117 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/Chacha.jpg HTTP/1.1 200 109379 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000159.jpg HTTP/1.1 200 161657 *When looking at the original log data outside of HDFS with the os version of the tail command we see the following* [training@localhost hands-on]$ hadoop fs -get weblog/access_log ./ [training@localhost hands-on]$ tail access_log 10.190.174.142 - - [03/Dec/2011:13:28:06 -0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 10.190.174.142 - - [03/Dec/2011:13:28:08 -0800] GET /assets/js/javascript_combined.js HTTP/1.1 200 20404 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /assets/img/home-logo.png HTTP/1.1 200 3892 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/019.jpg HTTP/1.1 200 74446 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/g_still_04.jpg HTTP/1.1 200 761555 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/07082218.jpg HTTP/1.1 200 154609 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000163.jpg HTTP/1.1 200 60117 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/Chacha.jpg HTTP/1.1 200 109379 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000159.jpg HTTP/1.1 200 161657 When using non ip data seperated by periods, it gets even worse and even more data is masked? (same data subtituting names for IP octects). Note we loose the first line well into the URI string? * [training@localhost hands-on]$ hadoop fs -tail weblog/test_log s/javascript_combined.js HTTP/1.1 200 20404 larry.billy.will.amy - - [03/Dec/2011:13:28:09 -0800] GET /assets/img/home-logo.png HTTP/1.1 200 3892 larry.billy.will.amy - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/019.jpg HTTP/1.1 200 74446 larry.billy.will.amy - - [03/Dec/2011:13:28:larry.-0800] GET /images/filmmediablock/360/g_still_04.jpg HTTP/1.1 200 761555 larry.billy.will.amy - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/07082218.jpg HTTP/1.1 200 154609 larry.billy.will.amy - - [03/Dec/2011:13:28:larry.-0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 larry.billy.will.amy - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000163.jpg HTTP/1.1 200 60117 larry.billy.will.amy - - [03/Dec/2011:13:28:larry.-0800] GET /images/filmmediablock/360/Chacha.jpg HTTP/1.1 200 larry.379 larry.billy.will.amy - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000159.jpg HTTP/1.1 200 161657 * and verifying what we are looking at in normal tail matches - note the first line is not
[jira] [Updated] (HDFS-4829) Strange loss of data displayed in hadoop fs -tail command
[ https://issues.apache.org/jira/browse/HDFS-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-4829: -- Issue Type: Improvement (was: Bug) Strange loss of data displayed in hadoop fs -tail command - Key: HDFS-4829 URL: https://issues.apache.org/jira/browse/HDFS-4829 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.0.0-alpha Environment: OS Centos 6.3 (on Intel Core2 Duo, VMware Player VM running under windows 7) Testing on both 2.0.0-cdh4.1.1 and 2.0.0-cdh4.1.2 Reporter: Todd Grayson Priority: Minor Strange behavior of the hadoop fs -tail command - its default for output seems to be 9 lines of output vs 10 lines of output in the OS version of the command (minor issue). The strange thing (bug behavior?) appears to drop the initial octect from an IP address when examining a file over HDFS. [training@localhost hands-on]$ hadoop fs -tail weblog/access_log .190.174.142 - - [03/Dec/2011:13:28:08 -0800] GET /assets/js/javascript_combined.js HTTP/1.1 200 20404 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /assets/img/home-logo.png HTTP/1.1 200 3892 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/019.jpg HTTP/1.1 200 74446 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/g_still_04.jpg HTTP/1.1 200 761555 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/07082218.jpg HTTP/1.1 200 154609 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000163.jpg HTTP/1.1 200 60117 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/Chacha.jpg HTTP/1.1 200 109379 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000159.jpg HTTP/1.1 200 161657 *When looking at the original log data outside of HDFS with the os version of the tail command we see the following* [training@localhost hands-on]$ hadoop fs -get weblog/access_log ./ [training@localhost hands-on]$ tail access_log 10.190.174.142 - - [03/Dec/2011:13:28:06 -0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 10.190.174.142 - - [03/Dec/2011:13:28:08 -0800] GET /assets/js/javascript_combined.js HTTP/1.1 200 20404 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /assets/img/home-logo.png HTTP/1.1 200 3892 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/019.jpg HTTP/1.1 200 74446 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/g_still_04.jpg HTTP/1.1 200 761555 10.190.174.142 - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/07082218.jpg HTTP/1.1 200 154609 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000163.jpg HTTP/1.1 200 60117 10.190.174.142 - - [03/Dec/2011:13:28:10 -0800] GET /images/filmmediablock/360/Chacha.jpg HTTP/1.1 200 109379 10.190.174.142 - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000159.jpg HTTP/1.1 200 161657 When using non ip data seperated by periods, it gets even worse and even more data is masked? (same data subtituting names for IP octects). Note we loose the first line well into the URI string? * [training@localhost hands-on]$ hadoop fs -tail weblog/test_log s/javascript_combined.js HTTP/1.1 200 20404 larry.billy.will.amy - - [03/Dec/2011:13:28:09 -0800] GET /assets/img/home-logo.png HTTP/1.1 200 3892 larry.billy.will.amy - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/019.jpg HTTP/1.1 200 74446 larry.billy.will.amy - - [03/Dec/2011:13:28:larry.-0800] GET /images/filmmediablock/360/g_still_04.jpg HTTP/1.1 200 761555 larry.billy.will.amy - - [03/Dec/2011:13:28:09 -0800] GET /images/filmmediablock/360/07082218.jpg HTTP/1.1 200 154609 larry.billy.will.amy - - [03/Dec/2011:13:28:larry.-0800] GET /images/filmpics//2229/GOEMON-NUKI-000163.jpg HTTP/1.1 200 184976 larry.billy.will.amy - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000163.jpg HTTP/1.1 200 60117 larry.billy.will.amy - - [03/Dec/2011:13:28:larry.-0800] GET /images/filmmediablock/360/Chacha.jpg HTTP/1.1 200 larry.379 larry.billy.will.amy - - [03/Dec/2011:13:28:11 -0800] GET /images/filmmediablock/360/GOEMON-NUKI-000159.jpg HTTP/1.1 200 161657 * and verifying what we are looking at in normal tail matches - note the first line is not
[jira] [Resolved] (HDFS-5740) getmerge file system shell command needs error message for user error
[
https://issues.apache.org/jira/browse/HDFS-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved HDFS-5740.
---
Resolution: Not a Problem
This is no longer an issue on branch-2 and trunk today. The command accepts a
collection of files now, and prepares the output accordingly.
getmerge file system shell command needs error message for user error
-
Key: HDFS-5740
URL: https://issues.apache.org/jira/browse/HDFS-5740
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 1.1.2
Environment: {noformat}[jpfuntner@h58 tmp]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.0 (Santiago)
[jpfuntner@h58 tmp]$ hadoop version
Hadoop 1.1.2.21
Subversion -r
Compiled by jenkins on Thu Jan 10 03:38:39 PST 2013
From source with checksum ce0aa0de785f572347f1afee69c73861{noformat}
Reporter: John Pfuntner
Priority: Minor
I naively tried a {{getmerge}} operation but it didn't seem to do anything
and there was no error message:
{noformat}[jpfuntner@h58 tmp]$ hadoop fs -mkdir /user/jpfuntner/tmp
[jpfuntner@h58 tmp]$ num=0; while [ $num -lt 5 ]; do echo file$num | hadoop
fs -put - /user/jpfuntner/tmp/file$num; let num=num+1; done
[jpfuntner@h58 tmp]$ ls -A
[jpfuntner@h58 tmp]$ hadoop fs -getmerge /user/jpfuntner/tmp/file* files.txt
[jpfuntner@h58 tmp]$ ls -A
[jpfuntner@h58 tmp]$ hadoop fs -ls /user/jpfuntner/tmp
Found 5 items
-rw--- 3 jpfuntner hdfs 6 2014-01-08 17:37
/user/jpfuntner/tmp/file0
-rw--- 3 jpfuntner hdfs 6 2014-01-08 17:37
/user/jpfuntner/tmp/file1
-rw--- 3 jpfuntner hdfs 6 2014-01-08 17:37
/user/jpfuntner/tmp/file2
-rw--- 3 jpfuntner hdfs 6 2014-01-08 17:37
/user/jpfuntner/tmp/file3
-rw--- 3 jpfuntner hdfs 6 2014-01-08 17:37
/user/jpfuntner/tmp/file4
[jpfuntner@h58 tmp]$ {noformat}
It was pointed out to me that I made a mistake and my source should have been
a directory not a set of regular files. It works if I use the directory:
{noformat}[jpfuntner@h58 tmp]$ hadoop fs -getmerge /user/jpfuntner/tmp/
files.txt
[jpfuntner@h58 tmp]$ ls -A
files.txt .files.txt.crc
[jpfuntner@h58 tmp]$ cat files.txt
file0
file1
file2
file3
file4
[jpfuntner@h58 tmp]$ {noformat}
I think the {{getmerge}} command should issue an error message to let the
user know they made a mistake.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HDFS-2360) Ugly stacktrace when quota exceeds
[ https://issues.apache.org/jira/browse/HDFS-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer reopened HDFS-2360: OK, then let me re-open it. Having oodles of useless stack trace here is *incredibly* user-unfriendly. Users do miss this message very very often because, believe it or not, they aren't Java programmers who are used to reading these things. Ugly stacktrace when quota exceeds -- Key: HDFS-2360 URL: https://issues.apache.org/jira/browse/HDFS-2360 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Affects Versions: 0.23.0 Reporter: Rajit Saha Priority: Minor Will it be better to catch the exception and throw a small reasonable messege to user when they exceed quota? $hdfs dfs -mkdir testDir $hdfs dfsadmin -setSpaceQuota 191M testDir $hdfs dfs -count -q testDir none inf 200278016 2002780161 0 0 hdfs://NN hostname:port/user/hdfsqa/testDir $hdfs dfs -put /etc/passwd /user/hadoopqa/testDir 11/09/19 08:08:15 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/hdfsqa/testDir is exceeded: quota=191.0m diskspace consumed=768.0m at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:159) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1609) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1383) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:370) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1681) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1476) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:389) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1496) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1492) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1490) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1100) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:972) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454) Caused by: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/hdfsqa/testDir is exceeded: quota=191.0m diskspace consumed=768.0m at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:159) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1609) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1383) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:370) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1681) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1476) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:389) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at
[jira] [Resolved] (HDFS-147) Stack trace on spaceQuota excced .
[
https://issues.apache.org/jira/browse/HDFS-147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDFS-147.
-
Resolution: Duplicate
Stack trace on spaceQuota excced .
--
Key: HDFS-147
URL: https://issues.apache.org/jira/browse/HDFS-147
Project: Hadoop HDFS
Issue Type: Bug
Environment: All
Reporter: Ravi Phulari
Assignee: Xiaoyu Yao
Labels: newbie
Currently disk space quota exceed exception spits out stack trace . It's
better to show error message instead of stack trace .
{code}
somehost:Hadoop guesti$ bin/hdfs dfsadmin -setSpaceQuota 2 2344
somehost:Hadoop guest$ bin/hadoop fs -put conf 2344
09/06/19 16:44:30 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.hdfs.protocol.QuotaExceededException:
org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
/user/guest/2344 is exceeded: namespace quota=-1 file count=4, diskspace
quota=2 diskspace=67108864
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
..
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-4494) Confusing exception for unresolvable hdfs host with security enabled
[
https://issues.apache.org/jira/browse/HDFS-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HDFS-4494:
--
Fix Version/s: 2.6.0
Confusing exception for unresolvable hdfs host with security enabled
Key: HDFS-4494
URL: https://issues.apache.org/jira/browse/HDFS-4494
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Priority: Minor
Fix For: 2.6.0
{noformat}
$ hadoop fs -ls hdfs://unresolvable-host
ls: Can't replace _HOST pattern since client address is null
{noformat}
It's misleading because it's not even related to the client's address. It'd
be a bit more informative to see something like {{UnknownHostException:
unresolvable-host}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-1330) Make RPCs to DataNodes timeout
[ https://issues.apache.org/jira/browse/HDFS-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-1330: --- Fix Version/s: (was: 2.0.0-alpha) Make RPCs to DataNodes timeout -- Key: HDFS-1330 URL: https://issues.apache.org/jira/browse/HDFS-1330 Project: Hadoop HDFS Issue Type: New Feature Components: datanode Affects Versions: 0.22.0, 0.23.0 Reporter: Hairong Kuang Assignee: John George Fix For: 0.22.0, 0.23.0 Attachments: HADOOP-6889-fortrunk-2.patch, hdfsRpcTimeout.patch This jira aims to make client/datanode or datanode/datanode RPC to have a timeout of DataNode#socketTimeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-2333) HDFS-2284 introduced 2 findbugs warnings on trunk
[ https://issues.apache.org/jira/browse/HDFS-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-2333: --- Fix Version/s: (was: 2.0.0-alpha) HDFS-2284 introduced 2 findbugs warnings on trunk - Key: HDFS-2333 URL: https://issues.apache.org/jira/browse/HDFS-2333 Project: Hadoop HDFS Issue Type: Bug Reporter: Ivan Kelly Assignee: Tsz Wo Nicholas Sze Fix For: 0.20.205.0, 0.23.0 Attachments: HDFS-2333.diff, h2333_20110914.patch, h2333_20110914b.patch, h2333_20110915.patch, h2333_20110915_0.20s.patch When HDFS-2284 was submitted it made DFSOutputStream public which triggered two SC_START_IN_CTOR findbug warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2360) Ugly stacktrace when quota exceeds
[ https://issues.apache.org/jira/browse/HDFS-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363619#comment-14363619 ] Allen Wittenauer commented on HDFS-2360: I'm re-opening to get rid of the stack trace as well. I see that someone else has also duped that request to this issue as well. :) Ugly stacktrace when quota exceeds -- Key: HDFS-2360 URL: https://issues.apache.org/jira/browse/HDFS-2360 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Affects Versions: 0.23.0 Reporter: Rajit Saha Priority: Minor Will it be better to catch the exception and throw a small reasonable messege to user when they exceed quota? $hdfs dfs -mkdir testDir $hdfs dfsadmin -setSpaceQuota 191M testDir $hdfs dfs -count -q testDir none inf 200278016 2002780161 0 0 hdfs://NN hostname:port/user/hdfsqa/testDir $hdfs dfs -put /etc/passwd /user/hadoopqa/testDir 11/09/19 08:08:15 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/hdfsqa/testDir is exceeded: quota=191.0m diskspace consumed=768.0m at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:159) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1609) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1383) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:370) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1681) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1476) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:389) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1496) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1492) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1490) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1100) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:972) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454) Caused by: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/hdfsqa/testDir is exceeded: quota=191.0m diskspace consumed=768.0m at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:159) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1609) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1383) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:370) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1681) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1476) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:389) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at
[jira] [Commented] (HDFS-7929) inotify unable fetch pre-upgrade edit log segments once upgrade starts
[
https://issues.apache.org/jira/browse/HDFS-7929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363646#comment-14363646
]
Hadoop QA commented on HDFS-7929:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704832/HDFS-7929-001.patch
against trunk revision ed4e72a.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9900//console
This message is automatically generated.
inotify unable fetch pre-upgrade edit log segments once upgrade starts
--
Key: HDFS-7929
URL: https://issues.apache.org/jira/browse/HDFS-7929
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-7929-000.patch, HDFS-7929-001.patch
inotify is often used to periodically poll HDFS events. However, once an HDFS
upgrade has started, edit logs are moved to /previous on the NN, which is not
accessible. Moreover, once the upgrade is finalized /previous is currently
lost forever.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-4494) Confusing exception for unresolvable hdfs host with security enabled
[
https://issues.apache.org/jira/browse/HDFS-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HDFS-4494:
--
Target Version/s: (was: 3.0.0, 2.1.0-beta)
Confusing exception for unresolvable hdfs host with security enabled
Key: HDFS-4494
URL: https://issues.apache.org/jira/browse/HDFS-4494
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Priority: Minor
Fix For: 2.6.0
{noformat}
$ hadoop fs -ls hdfs://unresolvable-host
ls: Can't replace _HOST pattern since client address is null
{noformat}
It's misleading because it's not even related to the client's address. It'd
be a bit more informative to see something like {{UnknownHostException:
unresolvable-host}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-2290) Block with corrupt replica is not getting replicated
[ https://issues.apache.org/jira/browse/HDFS-2290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-2290: --- Fix Version/s: (was: 2.0.0-alpha) Block with corrupt replica is not getting replicated Key: HDFS-2290 URL: https://issues.apache.org/jira/browse/HDFS-2290 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Benoy Antony Labels: blockmanagement, replication Fix For: 0.22.0, 0.23.0 Attachments: HDFS-2290_0.22.patch, HDFS-2290_022.patch, HDFS-2290_022.patch, HDFS-2290_trunk.patch, HDFS-2290_trunk.patch A block has one replica marked as corrupt and two good ones. countNodes() correctly detects that there are only 2 live replicas, and fsck reports the block as under-replicated. But ReplicationMonitor never schedules replication of good replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2360) Ugly stacktrace when quota exceeds
[ https://issues.apache.org/jira/browse/HDFS-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363612#comment-14363612 ] Harsh J commented on HDFS-2360: --- [~aw] - I agree with that sentiment, but the previous state was the lack of message. Now we do have the message, like my comment example indicates. Is that still unclear, or is the reopen just to get rid of the stack trace WARN as well on clients? Ugly stacktrace when quota exceeds -- Key: HDFS-2360 URL: https://issues.apache.org/jira/browse/HDFS-2360 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Affects Versions: 0.23.0 Reporter: Rajit Saha Priority: Minor Will it be better to catch the exception and throw a small reasonable messege to user when they exceed quota? $hdfs dfs -mkdir testDir $hdfs dfsadmin -setSpaceQuota 191M testDir $hdfs dfs -count -q testDir none inf 200278016 2002780161 0 0 hdfs://NN hostname:port/user/hdfsqa/testDir $hdfs dfs -put /etc/passwd /user/hadoopqa/testDir 11/09/19 08:08:15 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/hdfsqa/testDir is exceeded: quota=191.0m diskspace consumed=768.0m at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:159) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1609) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1383) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:370) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1681) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1476) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:389) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1496) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1492) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1490) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1100) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:972) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454) Caused by: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/hdfsqa/testDir is exceeded: quota=191.0m diskspace consumed=768.0m at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:159) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1609) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1383) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:370) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1681) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1476) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:389) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[jira] [Updated] (HDFS-2197) Refactor RPC call implementations out of NameNode class
[
https://issues.apache.org/jira/browse/HDFS-2197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HDFS-2197:
---
Fix Version/s: (was: 2.0.0-alpha)
Refactor RPC call implementations out of NameNode class
---
Key: HDFS-2197
URL: https://issues.apache.org/jira/browse/HDFS-2197
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: HA branch (HDFS-1623), 0.23.0
Attachments: hdfs-2197-2.txt, hdfs-2197-3.txt, hdfs-2197-4.txt,
hdfs-2197-trunk-final.txt, hdfs-2197.txt
For HA, the NameNode will gain a bit of a state machine, to be able to
transition between standby and active states. This would be cleaner in the
code if the {{NameNode}} class were just a container for various services, as
discussed in HDFS-1974. It's also nice for testing, where it would become
easier to construct just the RPC handlers around a mock NameSystem, with no
HTTP server, for example.
This JIRA is to move all of the protocol implementations out of {{NameNode}}
into a separate {{NameNodeRPCServer}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7929) inotify unable fetch pre-upgrade edit log segments once upgrade starts
[
https://issues.apache.org/jira/browse/HDFS-7929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-7929:
Attachment: HDFS-7929-001.patch
Some pre-created edits log files in {{TestDFSUpgradeFromImage}} do not have
version numbers. 001 patch uses {{EditLogFileInputStream#nextValidOp}} to skip
those.
inotify unable fetch pre-upgrade edit log segments once upgrade starts
--
Key: HDFS-7929
URL: https://issues.apache.org/jira/browse/HDFS-7929
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-7929-000.patch, HDFS-7929-001.patch
inotify is often used to periodically poll HDFS events. However, once an HDFS
upgrade has started, edit logs are moved to /previous on the NN, which is not
accessible. Moreover, once the upgrade is finalized /previous is currently
lost forever.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-4494) Confusing exception for unresolvable hdfs host with security enabled
[
https://issues.apache.org/jira/browse/HDFS-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved HDFS-4494.
---
Resolution: Done
Target Version/s: 2.1.0-beta, 3.0.0 (was: 3.0.0, 2.1.0-beta)
This seems resolved now (as of 2.6.0):
{code}
[root@host ~]# hdfs getconf -confKey hadoop.security.authentication
kerberos
[root@host ~]# hadoop fs -ls hdfs://asdfsdfsdf/
-ls: java.net.UnknownHostException: asdfsdfsdf
Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
{code}
Marking as Done.
Confusing exception for unresolvable hdfs host with security enabled
Key: HDFS-4494
URL: https://issues.apache.org/jira/browse/HDFS-4494
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Priority: Minor
{noformat}
$ hadoop fs -ls hdfs://unresolvable-host
ls: Can't replace _HOST pattern since client address is null
{noformat}
It's misleading because it's not even related to the client's address. It'd
be a bit more informative to see something like {{UnknownHostException:
unresolvable-host}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-4494) Confusing exception for unresolvable hdfs host with security enabled
[
https://issues.apache.org/jira/browse/HDFS-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HDFS-4494:
---
Fix Version/s: (was: 2.6.0)
Confusing exception for unresolvable hdfs host with security enabled
Key: HDFS-4494
URL: https://issues.apache.org/jira/browse/HDFS-4494
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Priority: Minor
{noformat}
$ hadoop fs -ls hdfs://unresolvable-host
ls: Can't replace _HOST pattern since client address is null
{noformat}
It's misleading because it's not even related to the client's address. It'd
be a bit more informative to see something like {{UnknownHostException:
unresolvable-host}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-4290) Expose an event listener interface in DFSOutputStreams for block write pipeline status changes
[ https://issues.apache.org/jira/browse/HDFS-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HDFS-4290. --- Resolution: Later Specific problems/use-cases driving this need haven't been bought up in the past years. Resolving as Later for now. Expose an event listener interface in DFSOutputStreams for block write pipeline status changes -- Key: HDFS-4290 URL: https://issues.apache.org/jira/browse/HDFS-4290 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client Affects Versions: 2.0.0-alpha Reporter: Harsh J Priority: Minor I've noticed HBase periodically polls the current status of block replicas for its HLog files via the API presented by HDFS-826. It would perhaps be better for such clients if they could register a listener instead. The listener(s) can be sent an event in case things change in the last open block (due to DN fall but no replacement found, etc. cases). This would avoid having a periodic, parallel looped check in such clients and be more efficient. Just a thought :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-2232) TestHDFSCLI fails on 0.22 branch
[ https://issues.apache.org/jira/browse/HDFS-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-2232: --- Fix Version/s: (was: 2.0.0-alpha) TestHDFSCLI fails on 0.22 branch Key: HDFS-2232 URL: https://issues.apache.org/jira/browse/HDFS-2232 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Plamen Jeliazkov Priority: Blocker Fix For: 0.22.0, 0.23.0 Attachments: HDFS-2232-TRUNK.patch, HDFS-2232-TRUNK.patch, HDFS-2232-TRUNK.patch, HDFS-2232-TRUNK.patch, HDFS-2232.patch, HDFS-2232.patch, HDFS-2232.patch, HDFS-2232.patch, TEST-org.apache.hadoop.cli.TestHDFSCLI.txt, TEST-org.apache.hadoop.cli.TestHDFSCLI.txt, TestHDFSCLI-output.txt Several HDFS CLI tests fail on 0.22 branch. I can see 2 reasons: # Not generic enough regular expression for host names and paths. Similar to MAPREDUCE-2304. # Some command outputs have new-line in the end. # And some seem to produce [much] more output than expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-2331) Hdfs compilation fails
[
https://issues.apache.org/jira/browse/HDFS-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HDFS-2331:
---
Fix Version/s: (was: 2.0.0-alpha)
Hdfs compilation fails
--
Key: HDFS-2331
URL: https://issues.apache.org/jira/browse/HDFS-2331
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 0.20.205.0, 2.0.0-alpha
Reporter: Abhijit Suresh Shingate
Assignee: Abhijit Suresh Shingate
Fix For: 0.20.205.0, 0.23.0
Attachments: HDFS-2331-1.patch, HDFS-2331.patch
Original Estimate: 1h
Remaining Estimate: 1h
I am trying to perform complete build from trunk folder but the compilation
fails.
*Commandline:*
mvn clean install
*Error Message:*
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.
3.2:compile (default-compile) on project hadoop-hdfs: Compilation failure
[ERROR] \Hadoop\SVN\trunk\hadoop-hdfs-project\hadoop-hdfs\src\main\java\org
\apache\hadoop\hdfs\web\WebHdfsFileSystem.java:[209,21] type parameters of
TT
cannot be determined; no unique maximal instance exists for type variable T
with
upper bounds T,java.lang.Object
[ERROR] - [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
swit
ch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
rea
d the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureExc
eption
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
This is because of known reason
[http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6302954]
{code:title=WebHdfsFileSystem.java|borderStyle=solid}
Following is the code snippet with issue.
private T T run(final HttpOpParam.Op op, final Path fspath,
final Param?,?... parameters) throws IOException {
final HttpURLConnection conn = httpConnect(op, fspath, parameters);
validateResponse(op, conn);
try {
return jsonParse(conn.getInputStream());
} finally {
conn.disconnect();
}
}
{code}
Fix to the issue
{code:title=WebHdfsFileSystem.java|borderStyle=solid}
Following is the fix to the issue.
private T T run(final HttpOpParam.Op op, final Path fspath,
final Param?,?... parameters) throws IOException {
final HttpURLConnection conn = httpConnect(op, fspath, parameters);
validateResponse(op, conn);
try {
return WebHdfsFileSystem.TjsonParse(conn.getInputStream());
} finally {
conn.disconnect();
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-3101) cannot read empty file using webhdfs
[
https://issues.apache.org/jira/browse/HDFS-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HDFS-3101:
---
Fix Version/s: (was: 2.0.0-alpha)
cannot read empty file using webhdfs
Key: HDFS-3101
URL: https://issues.apache.org/jira/browse/HDFS-3101
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 0.23.1
Reporter: Zhanwei Wang
Assignee: Tsz Wo Nicholas Sze
Fix For: 1.0.2, 0.23.2
Attachments: h3101_20120315.patch, h3101_20120315_branch-1.patch
STEP:
1, create a new EMPTY file
2, read it using webhdfs.
RESULT:
expected: get a empty file
I got:
{RemoteException:{exception:IOException,javaClassName:java.io.IOException,message:Offset=0
out of the range [0, 0); OPEN, path=/testFile}}
First of all, [0, 0) is not a valid range, and I think read a empty file
should be OK.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-2002) Incorrect computation of needed blocks in getTurnOffTip()
[
https://issues.apache.org/jira/browse/HDFS-2002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HDFS-2002:
---
Fix Version/s: (was: 2.0.0-alpha)
Incorrect computation of needed blocks in getTurnOffTip()
-
Key: HDFS-2002
URL: https://issues.apache.org/jira/browse/HDFS-2002
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Plamen Jeliazkov
Labels: newbie
Fix For: 0.22.0, 0.23.0
Attachments: HADOOP-2002_TRUNK.patch, hdfs-2002.patch,
testsafemode.patch, testsafemode.patch
{{SafeModeInfo.getTurnOffTip()}} under-reports the number of blocks needed to
reach the safemode threshold.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-2012) Recurring failure of TestBalancer due to incorrect treatment of nodes whose utilization equals avgUtilization.
[ https://issues.apache.org/jira/browse/HDFS-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-2012: --- Fix Version/s: (was: 2.0.0-alpha) Recurring failure of TestBalancer due to incorrect treatment of nodes whose utilization equals avgUtilization. -- Key: HDFS-2012 URL: https://issues.apache.org/jira/browse/HDFS-2012 Project: Hadoop HDFS Issue Type: Bug Components: balancer mover, test Affects Versions: 0.22.0 Reporter: Aaron T. Myers Assignee: Uma Maheswara Rao G Priority: Blocker Fix For: 0.22.0 Attachments: HDFS-2012-0.22Branch.patch, HDFS-2012-Trunk.patch, HDFS-2012.patch, TestBalancerLog.html This has been failing on Hudson for the last two builds and fails on my local box as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-2360) Ugly stacktrace when quota exceeds
[ https://issues.apache.org/jira/browse/HDFS-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-2360: -- Resolution: Fixed Fix Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks again Arpit (and Allen). Failing test was unrelated again, so I went ahead and committed this to branch-2 and trunk. Ugly stacktrace when quota exceeds -- Key: HDFS-2360 URL: https://issues.apache.org/jira/browse/HDFS-2360 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 0.23.0 Reporter: Rajit Saha Assignee: Harsh J Priority: Minor Fix For: 2.8.0 Attachments: HDFS-2360.patch, HDFS-2360.patch Will it be better to catch the exception and throw a small reasonable messege to user when they exceed quota? $hdfs dfs -mkdir testDir $hdfs dfsadmin -setSpaceQuota 191M testDir $hdfs dfs -count -q testDir none inf 200278016 2002780161 0 0 hdfs://NN hostname:port/user/hdfsqa/testDir $hdfs dfs -put /etc/passwd /user/hadoopqa/testDir 11/09/19 08:08:15 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/hdfsqa/testDir is exceeded: quota=191.0m diskspace consumed=768.0m at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:159) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1609) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1383) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:370) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1681) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1476) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:389) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1496) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1492) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1490) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1100) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:972) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454) Caused by: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/hdfsqa/testDir is exceeded: quota=191.0m diskspace consumed=768.0m at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:159) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1609) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1383) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:370) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1681) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1476) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:389) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at
[jira] [Assigned] (HDFS-7616) Change FSImage to support BlockGroup
[ https://issues.apache.org/jira/browse/HDFS-7616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze reassigned HDFS-7616: - Assignee: Takuya Fukudome (was: Tsz Wo Nicholas Sze) Sure, assigned to you. Thanks. Change FSImage to support BlockGroup Key: HDFS-7616 URL: https://issues.apache.org/jira/browse/HDFS-7616 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Takuya Fukudome We need to change FSImage for support BlockGroup and other new structures introduced for EC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2360) Ugly stacktrace when quota exceeds
[ https://issues.apache.org/jira/browse/HDFS-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364570#comment-14364570 ] Hudson commented on HDFS-2360: -- FAILURE: Integrated in Hadoop-trunk-Commit #7340 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7340/]) HDFS-2360. Ugly stacktrce when quota exceeds. (harsh) (harsh: rev 046521cd6511b7fc6d9478cb2bed90d8e75fca20) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Ugly stacktrace when quota exceeds -- Key: HDFS-2360 URL: https://issues.apache.org/jira/browse/HDFS-2360 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 0.23.0 Reporter: Rajit Saha Assignee: Harsh J Priority: Minor Fix For: 2.8.0 Attachments: HDFS-2360.patch, HDFS-2360.patch Will it be better to catch the exception and throw a small reasonable messege to user when they exceed quota? $hdfs dfs -mkdir testDir $hdfs dfsadmin -setSpaceQuota 191M testDir $hdfs dfs -count -q testDir none inf 200278016 2002780161 0 0 hdfs://NN hostname:port/user/hdfsqa/testDir $hdfs dfs -put /etc/passwd /user/hadoopqa/testDir 11/09/19 08:08:15 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/hdfsqa/testDir is exceeded: quota=191.0m diskspace consumed=768.0m at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:159) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1609) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1383) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:370) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1681) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1476) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:389) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1496) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1492) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1490) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1100) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:972) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454) Caused by: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/hdfsqa/testDir is exceeded: quota=191.0m diskspace consumed=768.0m at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:159) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1609) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1383) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:370) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1681) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1476) at
[jira] [Commented] (HDFS-5522) Datanode disk error check may be incorrectly skipped
[
https://issues.apache.org/jira/browse/HDFS-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364604#comment-14364604
]
Todd Lipcon commented on HDFS-5522:
---
I know this has been closed for a while, but wanted to get some clarification:
{quote}
If I/O hangs for a long time, network read/write may timeout first and the peer
may close the connection. Although the error was caused by a faulty local disk,
disk check is not being carried out in this case
{quote}
It seems like this JIRA applied a rather heavy hammer for a specific case that
could be better identified in another way. After applying this patch, it seems
that DNs will run checkDiskError when any other node experiences an issue. So,
if one node is down (eg due to a rolling restart or a crash) all of the other
nodes are very soon running checkDiskError for no particularly good reason.
Coupled with HDFS-7489, this failure can also cascade.
Can you describe in more detail the scenario you were facing that inspired this
JIRA? Would it not make more sense to actually look for the underlying symptom
(by adding a timer around the I/O, perhaps?) and running checkDiskError in the
specific scenarios we're looking for, rather than all network errors?
Datanode disk error check may be incorrectly skipped
Key: HDFS-5522
URL: https://issues.apache.org/jira/browse/HDFS-5522
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.23.9, 2.2.0
Reporter: Kihwal Lee
Assignee: Rushabh S Shah
Fix For: 2.5.0
Attachments: HDFS-5522-v2.patch, HDFS-5522-v3.patch, HDFS-5522.patch
After HDFS-4581 and HDFS-4699, {{checkDiskError()}} is not called when
network errors occur during processing data node requests. This appears to
create problems when a disk is having problems, but not failing I/O soon.
If I/O hangs for a long time, network read/write may timeout first and the
peer may close the connection. Although the error was caused by a faulty
local disk, disk check is not being carried out in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7930) commitBlockSynchronization() does not remove locations
[
https://issues.apache.org/jira/browse/HDFS-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364621#comment-14364621
]
Konstantin Shvachko commented on HDFS-7930:
---
Looks like the right fix. Few suggestions:
# Merge test cases {{testTruncateWithDataNodesRestart()}} and
{{testTruncateWithDataNodesRestart1()}} into one. The latter seems to be a more
detailed case of the former.
# Also you should use {{restartDataNode()}} with {{expireOnNN == true}} now
that HDFS-7886 is in.
# Remove misleading comment in {{FSN.commitBlockSync()}}
{{// There should be no locations in the blockManager till now }}
# Few improvements to {{markBlockReplicasAsCorrupt()}}
#- {{excludedStorages}} should be {{newStorages}} or {{committedStorages}}
#- Better keep the parameter declaration on a single line
#- {{toMark}} could be {{isCorrupt}}
#- Just {{else return;}} when the genStamp and the length are the same as the
old ones.
#- Redundant check {{excludedStorages.length 0}}
commitBlockSynchronization() does not remove locations
--
Key: HDFS-7930
URL: https://issues.apache.org/jira/browse/HDFS-7930
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.7.0
Reporter: Konstantin Shvachko
Assignee: Yi Liu
Priority: Blocker
Attachments: HDFS-7930.001.patch
When {{commitBlockSynchronization()}} has less {{newTargets}} than in the
original block it does not remove unconfirmed locations. This results in that
the the block stores locations of different lengths or genStamp (corrupt).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7717) Erasure Coding: provide a tool for convert files between replication and erasure coding
[ https://issues.apache.org/jira/browse/HDFS-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364641#comment-14364641 ] Kai Sasaki commented on HDFS-7717: -- [~jingzhao] Hello, Jing. I'd like to work on this issue. Could you assign it to me? Thank you. Erasure Coding: provide a tool for convert files between replication and erasure coding --- Key: HDFS-7717 URL: https://issues.apache.org/jira/browse/HDFS-7717 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Jing Zhao We need a tool to do offline conversion between replication and erasure coding. The tool itself can either utilize MR just like the current distcp, or act like the balancer/mover. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7864) Erasure Coding: Update safemode calculation for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-7864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364536#comment-14364536
]
GAO Rui commented on HDFS-7864:
---
[~jingzhao]
Thanks for your comment, Jing!
I have updated a new patch which used {{getDataBlockNum()}} to obtain the
minimal safe number for striped block. And I also modified code style according
to code convention as possible as I can. Please let me know if there are any
future problems.
Please review the patch when you are free, thank you very much!
And can you recommend another Jira for me to work on? Thank you!
Erasure Coding: Update safemode calculation for striped blocks
--
Key: HDFS-7864
URL: https://issues.apache.org/jira/browse/HDFS-7864
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: GAO Rui
Attachments: HDFS-7864.1.patch, HDFS-7864.2.patch, HDFS-7864.3.patch
We need to update the safemode calculation for striped blocks. Specifically,
each striped block now consists of multiple data/parity blocks stored in
corresponding DataNodes. The current code's calculation is thus inconsistent:
each striped block is only counted as 1 expected block, while each of its
member block may increase the number of received blocks by 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7829) Code clean up for LocatedBlock
[
https://issues.apache.org/jira/browse/HDFS-7829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-7829:
---
Attachment: HDFS-7829.4.patch
Thank you for your review, Jing. I thought and changed my patch:
1. I removed some constructors:{{LocatedBlock(ExtendedBlock, DatanodeInfo[],
long, boolean)}},{{LocatedBlock(ExtendedBlock, DatanodeStorageInfo[])}} and
changed callers.
And I kept {{LocatedBlock(ExtendedBlock b, DatanodeInfo[], String[],
StorageType[])}}.
2. Since {{setStartOffset}} and {{setCorrupt}} are used by tests out of the
package, I set public to them.
3. I set final to {{storageIDs}} and {{storageTypes}}.
How does that look?
Code clean up for LocatedBlock
--
Key: HDFS-7829
URL: https://issues.apache.org/jira/browse/HDFS-7829
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Jing Zhao
Assignee: Takanobu Asanuma
Priority: Minor
Attachments: HDFS-7829.1.patch, HDFS-7829.2.patch, HDFS-7829.3.patch,
HDFS-7829.4.patch
We can do some code cleanup for {{LocatedBlock}}, including:
# Using a simple Builder pattern to avoid multiple constructors
# Setting data fields like {{corrupt}} and {{offset}} to final
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7864) Erasure Coding: Update safemode calculation for striped blocks
[ https://issues.apache.org/jira/browse/HDFS-7864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] GAO Rui updated HDFS-7864: -- Attachment: HDFS-7864.3.patch Erasure Coding: Update safemode calculation for striped blocks -- Key: HDFS-7864 URL: https://issues.apache.org/jira/browse/HDFS-7864 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: GAO Rui Attachments: HDFS-7864.1.patch, HDFS-7864.2.patch, HDFS-7864.3.patch We need to update the safemode calculation for striped blocks. Specifically, each striped block now consists of multiple data/parity blocks stored in corresponding DataNodes. The current code's calculation is thus inconsistent: each striped block is only counted as 1 expected block, while each of its member block may increase the number of received blocks by 1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
