[jira] [Updated] (HDFS-6934) Move checksum computation off the hot path when writing to RAM disk
[
https://issues.apache.org/jira/browse/HDFS-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-6934:
Attachment: HDFS-6934.4.patch
Thank you for reviewing, Xiaoyu. Lots of good suggestions. Here is patch v4,
with the following changes.
{{FSOutputSummer}}: Used the common {{getChecksumSize}} method in more places
like you suggested.
{{DataChecksum}}: Added whitespace between operators.
{{DFSClient}}: Fixed minor Findbugs warning identified in last Jenkins run.
{{DFSInputStream}}: In {{getBestNodeDNAddrPair}}, moved storage types outside
the loop and added comment explaining indexed lookup of storage type.
{{LazyWriterTestCase}}: There was a lot of code duplication for helper methods
in {{TestLazyPersistFiles}} and {{TestScrLazyPersistFiles}}. I took the
opportunity to refactor into a new abstract base class that they can share.
{{TestScrLazyPersistFiles}}: The new tests are now in here.
bq. If you put Line 582~583 before Line 574, you can use...
I don't think we can make this change. The two conditions are logically
slightly different. On line 574:
{code}
if (checksumReceivedLen == 0 && !streams.isTransientStorage()) {
{code}
On line 582:
{code}
final boolean shouldNotWriteChecksum = checksumReceivedLen == 0
&& streams.isTransientStorage();
{code}
Note the negation of {{isTransientStorage()}} in the first one. The first case
is for detecting if a write to non-transient storage did not send a checksum,
and therefore it needs to be calculated here. The second case is for detecting
if this is a write to transient storage, and therefore the checksum needs to be
skipped entirely.
bq. I notice the test timeout attribute is removed. Is there a particular
reason for that?
The patch is using the {{Timeout}} JUnit rule to specify a timeout applied
across every test case in one place instead of specifying the timeout in each
{{Test}} annotation. We've been using this approach where possible in new
tests, because it avoids duplication of the timeout values and helps prevent
introducing new tests that forgot to specify a timeout. In the current version
of the patch, the {{Timeout}} rule is in the abstract base class, which covers
it for all tests in both subclasses.
> Move checksum computation off the hot path when writing to RAM disk
> ---
>
> Key: HDFS-6934
> URL: https://issues.apache.org/jira/browse/HDFS-6934
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Chris Nauroth
> Attachments: HDFS-6934.3.patch, HDFS-6934.4.patch,
> h6934_20141003b.patch, h6934_20141005.patch
>
>
> Since local RAM is considered reliable we can avoid writing checksums on the
> hot path when replicas are being written to a local RAM disk.
> The checksum can be computed by the lazy writer when moving replicas to disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7278) Add a command that allows sysadmins to manually trigger full block reports from a DN
[
https://issues.apache.org/jira/browse/HDFS-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183901#comment-14183901
]
Hadoop QA commented on HDFS-7278:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677022/HDFS-7278.005.patch
against trunk revision 683897f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8538//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8538//console
This message is automatically generated.
> Add a command that allows sysadmins to manually trigger full block reports
> from a DN
>
>
> Key: HDFS-7278
> URL: https://issues.apache.org/jira/browse/HDFS-7278
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7278.002.patch, HDFS-7278.003.patch,
> HDFS-7278.004.patch, HDFS-7278.005.patch
>
>
> We should add a command that allows sysadmins to manually trigger full block
> reports from a DN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7281) Missing block is marked as corrupted block
[
https://issues.apache.org/jira/browse/HDFS-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183900#comment-14183900
]
Hadoop QA commented on HDFS-7281:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677079/HDFS-7281-2.patch
against trunk revision 683897f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8539//console
This message is automatically generated.
> Missing block is marked as corrupted block
> --
>
> Key: HDFS-7281
> URL: https://issues.apache.org/jira/browse/HDFS-7281
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-7281-2.patch, HDFS-7281.patch
>
>
> In the situation where the block lost all its replicas, fsck shows the block
> is missing as well as corrupted. Perhaps it is better not to mark the block
> corrupted in this case. The reason it is marked as corrupted is
> numCorruptNodes == numNodes == 0 in the following code.
> {noformat}
> BlockManager
> final boolean isCorrupt = numCorruptNodes == numNodes;
> {noformat}
> Would like to clarify if it is the intent to mark missing block as corrupted
> or it is just a bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7281) Missing block is marked as corrupted block
[
https://issues.apache.org/jira/browse/HDFS-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-7281:
--
Attachment: HDFS-7281-2.patch
Thanks, Yongjun. Here is the updated patch to address your comment.
> Missing block is marked as corrupted block
> --
>
> Key: HDFS-7281
> URL: https://issues.apache.org/jira/browse/HDFS-7281
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-7281-2.patch, HDFS-7281.patch
>
>
> In the situation where the block lost all its replicas, fsck shows the block
> is missing as well as corrupted. Perhaps it is better not to mark the block
> corrupted in this case. The reason it is marked as corrupted is
> numCorruptNodes == numNodes == 0 in the following code.
> {noformat}
> BlockManager
> final boolean isCorrupt = numCorruptNodes == numNodes;
> {noformat}
> Would like to clarify if it is the intent to mark missing block as corrupted
> or it is just a bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5928) show namespace and namenode ID on NN dfshealth page
[
https://issues.apache.org/jira/browse/HDFS-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183878#comment-14183878
]
Hadoop QA commented on HDFS-5928:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677054/HDFS-5928.007.patch
against trunk revision 683897f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8537//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8537//console
This message is automatically generated.
> show namespace and namenode ID on NN dfshealth page
> ---
>
> Key: HDFS-5928
> URL: https://issues.apache.org/jira/browse/HDFS-5928
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siqi Li
>Assignee: Siqi Li
> Attachments: HDFS-5928.007.patch, HDFS-5928.v2.patch,
> HDFS-5928.v3.patch, HDFS-5928.v4.patch, HDFS-5928.v5.patch,
> HDFS-5928.v6.patch, HDFS－5928.v1.patch, screenshot-1.png
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7017) Implement OutputStream for libhdfs3
[ https://issues.apache.org/jira/browse/HDFS-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183876#comment-14183876 ] Zhanwei Wang commented on HDFS-7017: In HDFS-7017-pnative.002.patch, do not throw exception in OutputStrean interface, return Status instead. > Implement OutputStream for libhdfs3 > --- > > Key: HDFS-7017 > URL: https://issues.apache.org/jira/browse/HDFS-7017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Zhanwei Wang >Assignee: Zhanwei Wang > Attachments: HDFS-7017-pnative.002.patch, HDFS-7017.patch > > > Implement pipeline and OutputStream C++ interface -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7017) Implement OutputStream for libhdfs3
[ https://issues.apache.org/jira/browse/HDFS-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhanwei Wang updated HDFS-7017: --- Attachment: HDFS-7017-pnative.002.patch > Implement OutputStream for libhdfs3 > --- > > Key: HDFS-7017 > URL: https://issues.apache.org/jira/browse/HDFS-7017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Zhanwei Wang >Assignee: Zhanwei Wang > Attachments: HDFS-7017-pnative.002.patch, HDFS-7017.patch > > > Implement pipeline and OutputStream C++ interface -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7235) Can not decommission DN which has invalid block due to bad disk
[
https://issues.apache.org/jira/browse/HDFS-7235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183862#comment-14183862
]
Yongjun Zhang commented on HDFS-7235:
-
The test result shows no failure, except the log has the following in the end:
{code}
+ mv
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build@2/../patchprocess
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build@2/patchprocess
mv: cannot stat
'/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build@2/../patchprocess':
No such file or directory
Build step 'Execute shell' marked build as failure
Archiving artifacts
Description set: HDFS-7235
Recording test results
Publish JUnit test result report is waiting for a checkpoint on
PreCommit-HDFS-Build #8535
Finished: FAILURE
{code}
which happens to other tests from time to time. It appears to be a test
infrastructure issue.
> Can not decommission DN which has invalid block due to bad disk
> ---
>
> Key: HDFS-7235
> URL: https://issues.apache.org/jira/browse/HDFS-7235
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-7235.001.patch, HDFS-7235.002.patch,
> HDFS-7235.003.patch, HDFS-7235.004.patch, HDFS-7235.005.patch,
> HDFS-7235.006.patch
>
>
> When to decommission a DN, the process hangs.
> What happens is, when NN chooses a replica as a source to replicate data on
> the to-be-decommissioned DN to other DNs, it favors choosing this DN
> to-be-decommissioned as the source of transfer (see BlockManager.java).
> However, because of the bad disk, the DN would detect the source block to be
> transfered as invalidBlock with the following logic in FsDatasetImpl.java:
> {code}
> /** Does the block exist and have the given state? */
> private boolean isValid(final ExtendedBlock b, final ReplicaState state) {
> final ReplicaInfo replicaInfo = volumeMap.get(b.getBlockPoolId(),
> b.getLocalBlock());
> return replicaInfo != null
> && replicaInfo.getState() == state
> && replicaInfo.getBlockFile().exists();
> }
> {code}
> The reason that this method returns false (detecting invalid block) is
> because the block file doesn't exist due to bad disk in this case.
> The key issue we found here is, after DN detects an invalid block for the
> above reason, it doesn't report the invalid block back to NN, thus NN doesn't
> know that the block is corrupted, and keeps sending the data transfer request
> to the same DN to be decommissioned, again and again. This caused an infinite
> loop, so the decommission process hangs.
> Thanks [~qwertymaniac] for reporting the issue and initial analysis.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7279) Use netty to implement DatanodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183855#comment-14183855 ] Haohui Mai commented on HDFS-7279: -- The v1 patch implements kerberos and HTTPS support. It should have function parity for the implementation today. > Use netty to implement DatanodeWebHdfsMethods > - > > Key: HDFS-7279 > URL: https://issues.apache.org/jira/browse/HDFS-7279 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, webhdfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-7279.000.patch, HDFS-7279.001.patch > > > Currently the DN implements all related webhdfs functionality using jetty. As > the current jetty version the DN used (jetty 6) lacks of fine-grained buffer > and connection management, DN often suffers from long latency and OOM when > its webhdfs component is under sustained heavy load. > This jira proposes to implement the webhdfs component in DN using netty, > which can be more efficient and allow more finer-grain controls on webhdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7279) Use netty to implement DatanodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-7279: - Attachment: HDFS-7279.001.patch > Use netty to implement DatanodeWebHdfsMethods > - > > Key: HDFS-7279 > URL: https://issues.apache.org/jira/browse/HDFS-7279 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, webhdfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-7279.000.patch, HDFS-7279.001.patch > > > Currently the DN implements all related webhdfs functionality using jetty. As > the current jetty version the DN used (jetty 6) lacks of fine-grained buffer > and connection management, DN often suffers from long latency and OOM when > its webhdfs component is under sustained heavy load. > This jira proposes to implement the webhdfs component in DN using netty, > which can be more efficient and allow more finer-grain controls on webhdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7173) Only keep successfully loaded volumes in the configuration.
[
https://issues.apache.org/jira/browse/HDFS-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183838#comment-14183838
]
Hadoop QA commented on HDFS-7173:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677031/HDFS-7173.003.combo.patch
against trunk revision 683897f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8535//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8535//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8535//console
This message is automatically generated.
> Only keep successfully loaded volumes in the configuration.
> ---
>
> Key: HDFS-7173
> URL: https://issues.apache.org/jira/browse/HDFS-7173
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0, 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7173.000.combo.patch, HDFS-7173.000.patch,
> HDFS-7173.001.combo.patch, HDFS-7173.001.patch, HDFS-7173.002.combo.patch,
> HDFS-7173.002.patch, HDFS-7173.003.combo.patch, HDFS-7173.003.patch
>
>
> Hot swapping data volumes might fail. The user should be able to fix the
> failed volumes and disks, then ask the {{DataNode}} to retry the previously
> failed volumes.
> To attempt to reload the failed volume again on the same directory, this
> failed directory must not be presented in the {{Configuration}} object that
> {{DataNode has}}. Therefore, it should only put successfully loaded volumes
> into the {{Configuration}} object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7278) Add a command that allows sysadmins to manually trigger full block reports from a DN
[ https://issues.apache.org/jira/browse/HDFS-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183811#comment-14183811 ] Colin Patrick McCabe commented on HDFS-7278: It's really odd that the test run for https://issues.apache.org/jira/secure/attachment/12677022/HDFS-7278.005.patch says that the patch "does not include any new or modified tests." You can clearly see that the patch includes a new test, hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java. I am re-triggering jenkins. > Add a command that allows sysadmins to manually trigger full block reports > from a DN > > > Key: HDFS-7278 > URL: https://issues.apache.org/jira/browse/HDFS-7278 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-7278.002.patch, HDFS-7278.003.patch, > HDFS-7278.004.patch, HDFS-7278.005.patch > > > We should add a command that allows sysadmins to manually trigger full block > reports from a DN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7235) Can not decommission DN which has invalid block due to bad disk
[
https://issues.apache.org/jira/browse/HDFS-7235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183809#comment-14183809
]
Hadoop QA commented on HDFS-7235:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677035/HDFS-7235.006.patch
against trunk revision 683897f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-hdfs-project/hadoop-hdfs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8536//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8536//console
This message is automatically generated.
> Can not decommission DN which has invalid block due to bad disk
> ---
>
> Key: HDFS-7235
> URL: https://issues.apache.org/jira/browse/HDFS-7235
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-7235.001.patch, HDFS-7235.002.patch,
> HDFS-7235.003.patch, HDFS-7235.004.patch, HDFS-7235.005.patch,
> HDFS-7235.006.patch
>
>
> When to decommission a DN, the process hangs.
> What happens is, when NN chooses a replica as a source to replicate data on
> the to-be-decommissioned DN to other DNs, it favors choosing this DN
> to-be-decommissioned as the source of transfer (see BlockManager.java).
> However, because of the bad disk, the DN would detect the source block to be
> transfered as invalidBlock with the following logic in FsDatasetImpl.java:
> {code}
> /** Does the block exist and have the given state? */
> private boolean isValid(final ExtendedBlock b, final ReplicaState state) {
> final ReplicaInfo replicaInfo = volumeMap.get(b.getBlockPoolId(),
> b.getLocalBlock());
> return replicaInfo != null
> && replicaInfo.getState() == state
> && replicaInfo.getBlockFile().exists();
> }
> {code}
> The reason that this method returns false (detecting invalid block) is
> because the block file doesn't exist due to bad disk in this case.
> The key issue we found here is, after DN detects an invalid block for the
> above reason, it doesn't report the invalid block back to NN, thus NN doesn't
> know that the block is corrupted, and keeps sending the data transfer request
> to the same DN to be decommissioned, again and again. This caused an infinite
> loop, so the decommission process hangs.
> Thanks [~qwertymaniac] for reporting the issue and initial analysis.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7173) Only keep successfully loaded volumes in the configuration.
[
https://issues.apache.org/jira/browse/HDFS-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183794#comment-14183794
]
Hadoop QA commented on HDFS-7173:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677015/HDFS-7173.002.combo.patch
against trunk revision 0ac6988.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8533//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8533//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8533//console
This message is automatically generated.
> Only keep successfully loaded volumes in the configuration.
> ---
>
> Key: HDFS-7173
> URL: https://issues.apache.org/jira/browse/HDFS-7173
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0, 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7173.000.combo.patch, HDFS-7173.000.patch,
> HDFS-7173.001.combo.patch, HDFS-7173.001.patch, HDFS-7173.002.combo.patch,
> HDFS-7173.002.patch, HDFS-7173.003.combo.patch, HDFS-7173.003.patch
>
>
> Hot swapping data volumes might fail. The user should be able to fix the
> failed volumes and disks, then ask the {{DataNode}} to retry the previously
> failed volumes.
> To attempt to reload the failed volume again on the same directory, this
> failed directory must not be presented in the {{Configuration}} object that
> {{DataNode has}}. Therefore, it should only put successfully loaded volumes
> into the {{Configuration}} object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7200) Rename libhdfs3 to libndfs++
[ https://issues.apache.org/jira/browse/HDFS-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183780#comment-14183780 ] Colin Patrick McCabe commented on HDFS-7200: [~aw], can you comment on this? I remember you felt strongly about the library name. I don't particularly mind what this is called, but I would like to avoid something confusingly similar to the existing "libhdfs". I also don't like the "\+\+" proposals because plus signs are a special character that creates problems in a bunch of places, and because this library will be usable from C as well as C\+\+. I still like the "libndfs" proposal but I'm open to other ideas. > Rename libhdfs3 to libndfs++ > > > Key: HDFS-7200 > URL: https://issues.apache.org/jira/browse/HDFS-7200 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HADOOP-10388 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-7200.001.patch > > > Since we generally agree that libhdfs3 is a sub-optimal name, let's call the > new library "libndfs++." -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7287) The OfflineImageViewer (OIV) can output invalid XML depending on the filename
[
https://issues.apache.org/jira/browse/HDFS-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183759#comment-14183759
]
Hadoop QA commented on HDFS-7287:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677005/HDFS-7287.1.patch
against trunk revision a52eb4b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestParallelRead
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8532//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8532//console
This message is automatically generated.
> The OfflineImageViewer (OIV) can output invalid XML depending on the filename
> -
>
> Key: HDFS-7287
> URL: https://issues.apache.org/jira/browse/HDFS-7287
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HDFS-7287.1.patch, HDFS-7287.patch
>
>
> If the filename contains a character which is invalid in XML,
> TextWriterImageVisitor.write() or PBImageXmlWriter.o() prints out the string
> unescaped. For us this was the character 0x0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7035) Make adding volume an atomic operation.
[
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183758#comment-14183758
]
Hadoop QA commented on HDFS-7035:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677004/HDFS-7035.009.patch
against trunk revision a52eb4b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1265 javac
compiler warnings (more than the trunk's current 1264 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.fs.viewfs.TestViewFsWithAcls
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestParallelRead
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8531//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8531//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8531//console
This message is automatically generated.
> Make adding volume an atomic operation.
> ---
>
> Key: HDFS-7035
> URL: https://issues.apache.org/jira/browse/HDFS-7035
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch,
> HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch,
> HDFS-7035.003.patch, HDFS-7035.003.patch, HDFS-7035.004.patch,
> HDFS-7035.005.patch, HDFS-7035.007.patch, HDFS-7035.008.patch,
> HDFS-7035.009.patch
>
>
> It refactors {{DataStorage}} and {{BlockPoolSliceStorage}} to reduce the
> duplicate code and supports atomic adding volume operations. Also it
> parallels loading data volume operation: each thread loads one volume.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5928) show namespace and namenode ID on NN dfshealth page
[ https://issues.apache.org/jira/browse/HDFS-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183739#comment-14183739 ] Haohui Mai commented on HDFS-5928: -- I found that the v6 patch does not display correctly in non-HA set up. Made trivial changes fix the bug in the v7 patch. [~l201514], does it look good to you? > show namespace and namenode ID on NN dfshealth page > --- > > Key: HDFS-5928 > URL: https://issues.apache.org/jira/browse/HDFS-5928 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Siqi Li >Assignee: Siqi Li > Attachments: HDFS-5928.007.patch, HDFS-5928.v2.patch, > HDFS-5928.v3.patch, HDFS-5928.v4.patch, HDFS-5928.v5.patch, > HDFS-5928.v6.patch, HDFS－5928.v1.patch, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-5928) show namespace and namenode ID on NN dfshealth page
[ https://issues.apache.org/jira/browse/HDFS-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5928: - Attachment: HDFS-5928.007.patch > show namespace and namenode ID on NN dfshealth page > --- > > Key: HDFS-5928 > URL: https://issues.apache.org/jira/browse/HDFS-5928 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Siqi Li >Assignee: Siqi Li > Attachments: HDFS-5928.007.patch, HDFS-5928.v2.patch, > HDFS-5928.v3.patch, HDFS-5928.v4.patch, HDFS-5928.v5.patch, > HDFS-5928.v6.patch, HDFS－5928.v1.patch, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7231) rollingupgrade needs some guard rails
[ https://issues.apache.org/jira/browse/HDFS-7231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183724#comment-14183724 ] Allen Wittenauer edited comment on HDFS-7231 at 10/24/14 11:30 PM: --- bq. (unable to continue) This is the part I've also been trying to get more information on as well. :D From what I've been told, the first few directories were converted to the new format but not all of them. This seems to imply that the DN process was brought down or crashed at some point in time. bq. This as you know is a recipe for disaster I can neither confirm nor deny this statement. ;) bq. Before you go on to 2.4.1, if you do finalize of upgrade what happens? It appears to work as expected. So this whole condition appears to trigger with non-finalized systems. was (Author: aw): bq. (unable to continue) This is the part I've also been trying to get more information on as well. :D From what I've been told, the first few directories were converted to the new format but not all of them. This seems to imply that the DN process was brought down at some point in time. bq. This as you know is a recipe for disaster I can neither confirm nor deny this statement. ;) bq. Before you go on to 2.4.1, if you do finalize of upgrade what happens? It appears to work as expected. So this whole condition appears to trigger with non-finalized systems. > rollingupgrade needs some guard rails > - > > Key: HDFS-7231 > URL: https://issues.apache.org/jira/browse/HDFS-7231 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Allen Wittenauer >Priority: Blocker > > See first comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7278) Add a command that allows sysadmins to manually trigger full block reports from a DN
[
https://issues.apache.org/jira/browse/HDFS-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183725#comment-14183725
]
Hadoop QA commented on HDFS-7278:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676993/HDFS-7278.004.patch
against trunk revision a52eb4b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
org.apache.hadoop.hdfs.TestDistributedFileSystem
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8529//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8529//console
This message is automatically generated.
> Add a command that allows sysadmins to manually trigger full block reports
> from a DN
>
>
> Key: HDFS-7278
> URL: https://issues.apache.org/jira/browse/HDFS-7278
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7278.002.patch, HDFS-7278.003.patch,
> HDFS-7278.004.patch, HDFS-7278.005.patch
>
>
> We should add a command that allows sysadmins to manually trigger full block
> reports from a DN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7231) rollingupgrade needs some guard rails
[ https://issues.apache.org/jira/browse/HDFS-7231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183724#comment-14183724 ] Allen Wittenauer edited comment on HDFS-7231 at 10/24/14 11:29 PM: --- bq. (unable to continue) This is the part I've also been trying to get more information on as well. :D From what I've been told, the first few directories were converted to the new format but not all of them. This seems to imply that the DN process was brought down at some point in time. bq. This as you know is a recipe for disaster I can neither confirm nor deny this statement. ;) bq. Before you go on to 2.4.1, if you do finalize of upgrade what happens? It appears to work as expected. So this whole condition appears to trigger with non-finalized systems. was (Author: aw): bq. (unable to continue) This is the part I've also been trying to get more information on as well. :D From what I've been told, the first few directories were converted to the new format but not all of them. This seems to imply that the DN process was brought down at some point in time. bq. This as you know is a recipe for disaster I can neither confirm or deny this statement. ;) bq. Before you go on to 2.4.1, if you do finalize of upgrade what happens? It appears to work as expected. So this whole condition appears to trigger with non-finalized systems. > rollingupgrade needs some guard rails > - > > Key: HDFS-7231 > URL: https://issues.apache.org/jira/browse/HDFS-7231 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Allen Wittenauer >Priority: Blocker > > See first comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7231) rollingupgrade needs some guard rails
[ https://issues.apache.org/jira/browse/HDFS-7231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183724#comment-14183724 ] Allen Wittenauer commented on HDFS-7231: bq. (unable to continue) This is the part I've also been trying to get more information on as well. :D From what I've been told, the first few directories were converted to the new format but not all of them. This seems to imply that the DN process was brought down at some point in time. bq. This as you know is a recipe for disaster I can neither confirm or deny this statement. ;) bq. Before you go on to 2.4.1, if you do finalize of upgrade what happens? It appears to work as expected. So this whole condition appears to trigger with non-finalized systems. > rollingupgrade needs some guard rails > - > > Key: HDFS-7231 > URL: https://issues.apache.org/jira/browse/HDFS-7231 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Allen Wittenauer >Priority: Blocker > > See first comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7014) Implement input streams and file system functionality
[ https://issues.apache.org/jira/browse/HDFS-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183716#comment-14183716 ] Colin Patrick McCabe edited comment on HDFS-7014 at 10/24/14 11:26 PM: --- Sorry, [~wheat9]. It looks like our comments collided (I didn't see your comment before committing). Since this was posted 2 days ago, I assumed you had looked at it. Can we simply file follow-up JIRAs for any issues we find? P.S. having this in will make it easier for us to work on HDFS-7022 and HDFS-7023. The output stream stuff has been moved to HDFS-7017 as you asked, so having this in unblocks that JIRA as well. was (Author: cmccabe): Sorry, [~wheat9]. It looks like our comments collided (I didn't see your comment before committing). Since this was posted 2 days ago, I assumed you had looked at it. Can we simply file follow-up JIRAs for any issues we find? > Implement input streams and file system functionality > - > > Key: HDFS-7014 > URL: https://issues.apache.org/jira/browse/HDFS-7014 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Zhanwei Wang >Assignee: Zhanwei Wang > Fix For: HDFS-6994 > > Attachments: 0001-HDFS-7014-001.patch, HDFS-7014-pnative.002.patch, > HDFS-7014-pnative.003.patch, HDFS-7014-pnative.004.patch, HDFS-7014.patch > > > Implement Client - Namenode RPC protocol and support Namenode HA. > Implement Client - Datanode RPC protocol. > Implement some basic server side class such as ExtendedBlock and LocatedBlock -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7014) Implement input streams and file system functionality
[ https://issues.apache.org/jira/browse/HDFS-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183716#comment-14183716 ] Colin Patrick McCabe commented on HDFS-7014: Sorry, [~wheat9]. It looks like our comments collided (I didn't see your comment before committing). Since this was posted 2 days ago, I assumed you had looked at it. Can we simply file follow-up JIRAs for any issues we find? > Implement input streams and file system functionality > - > > Key: HDFS-7014 > URL: https://issues.apache.org/jira/browse/HDFS-7014 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Zhanwei Wang >Assignee: Zhanwei Wang > Fix For: HDFS-6994 > > Attachments: 0001-HDFS-7014-001.patch, HDFS-7014-pnative.002.patch, > HDFS-7014-pnative.003.patch, HDFS-7014-pnative.004.patch, HDFS-7014.patch > > > Implement Client - Namenode RPC protocol and support Namenode HA. > Implement Client - Datanode RPC protocol. > Implement some basic server side class such as ExtendedBlock and LocatedBlock -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7014) Implement input streams and file system functionality
[ https://issues.apache.org/jira/browse/HDFS-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe resolved HDFS-7014. Resolution: Fixed Fix Version/s: HDFS-6994 > Implement input streams and file system functionality > - > > Key: HDFS-7014 > URL: https://issues.apache.org/jira/browse/HDFS-7014 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Zhanwei Wang >Assignee: Zhanwei Wang > Fix For: HDFS-6994 > > Attachments: 0001-HDFS-7014-001.patch, HDFS-7014-pnative.002.patch, > HDFS-7014-pnative.003.patch, HDFS-7014-pnative.004.patch, HDFS-7014.patch > > > Implement Client - Namenode RPC protocol and support Namenode HA. > Implement Client - Datanode RPC protocol. > Implement some basic server side class such as ExtendedBlock and LocatedBlock -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7014) Implement input streams and file system functionality
[ https://issues.apache.org/jira/browse/HDFS-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183705#comment-14183705 ] Haohui Mai commented on HDFS-7014: -- Please allow me to have a couple days to review the patch. Thanks. > Implement input streams and file system functionality > - > > Key: HDFS-7014 > URL: https://issues.apache.org/jira/browse/HDFS-7014 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Zhanwei Wang >Assignee: Zhanwei Wang > Attachments: 0001-HDFS-7014-001.patch, HDFS-7014-pnative.002.patch, > HDFS-7014-pnative.003.patch, HDFS-7014-pnative.004.patch, HDFS-7014.patch > > > Implement Client - Namenode RPC protocol and support Namenode HA. > Implement Client - Datanode RPC protocol. > Implement some basic server side class such as ExtendedBlock and LocatedBlock -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7014) Implement input streams and file system functionality
[ https://issues.apache.org/jira/browse/HDFS-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183701#comment-14183701 ] Colin Patrick McCabe commented on HDFS-7014: Thanks, Zhanwei. +1. > Implement input streams and file system functionality > - > > Key: HDFS-7014 > URL: https://issues.apache.org/jira/browse/HDFS-7014 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Zhanwei Wang >Assignee: Zhanwei Wang > Attachments: 0001-HDFS-7014-001.patch, HDFS-7014-pnative.002.patch, > HDFS-7014-pnative.003.patch, HDFS-7014-pnative.004.patch, HDFS-7014.patch > > > Implement Client - Namenode RPC protocol and support Namenode HA. > Implement Client - Datanode RPC protocol. > Implement some basic server side class such as ExtendedBlock and LocatedBlock -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7278) Add a command that allows sysadmins to manually trigger full block reports from a DN
[
https://issues.apache.org/jira/browse/HDFS-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183685#comment-14183685
]
Hadoop QA commented on HDFS-7278:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677022/HDFS-7278.005.patch
against trunk revision 683897f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
org.apache.hadoop.ha.TestZKFailoverControllerStress
The following test timeouts occurred in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.ha.TestZKFailoverControllerStress
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8534//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8534//console
This message is automatically generated.
> Add a command that allows sysadmins to manually trigger full block reports
> from a DN
>
>
> Key: HDFS-7278
> URL: https://issues.apache.org/jira/browse/HDFS-7278
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7278.002.patch, HDFS-7278.003.patch,
> HDFS-7278.004.patch, HDFS-7278.005.patch
>
>
> We should add a command that allows sysadmins to manually trigger full block
> reports from a DN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7035) Make adding volume an atomic operation.
[
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183684#comment-14183684
]
Lei (Eddy) Xu commented on HDFS-7035:
-
I am working on addressing the fix bug reports and related failure case
(TestDistributedFileSystem)
> Make adding volume an atomic operation.
> ---
>
> Key: HDFS-7035
> URL: https://issues.apache.org/jira/browse/HDFS-7035
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch,
> HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch,
> HDFS-7035.003.patch, HDFS-7035.003.patch, HDFS-7035.004.patch,
> HDFS-7035.005.patch, HDFS-7035.007.patch, HDFS-7035.008.patch,
> HDFS-7035.009.patch
>
>
> It refactors {{DataStorage}} and {{BlockPoolSliceStorage}} to reduce the
> duplicate code and supports atomic adding volume operations. Also it
> parallels loading data volume operation: each thread loads one volume.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7276) Limit the number of byte arrays used by DFSOutputStream
[ https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183683#comment-14183683 ] stack commented on HDFS-7276: - bq. ...It needs volatile. Oh, I see. You actually assign to a variable that is all uppercase. nit: Uppercase is usually a flag that variable is constant. Suggest you undo the uppercasing. > Limit the number of byte arrays used by DFSOutputStream > --- > > Key: HDFS-7276 > URL: https://issues.apache.org/jira/browse/HDFS-7276 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h7276_20141021.patch, h7276_20141022.patch, > h7276_20141023.patch, h7276_20141024.patch > > > When there are a lot of DFSOutputStream's writing concurrently, the number of > outstanding packets could be large. The byte arrays created by those packets > could occupy a lot of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6934) Move checksum computation off the hot path when writing to RAM disk
[
https://issues.apache.org/jira/browse/HDFS-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183675#comment-14183675
]
Hadoop QA commented on HDFS-6934:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676980/HDFS-6934.3.patch
against trunk revision 86ac0d4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.protocolPB.TestPBHelper
org.apache.hadoop.hdfs.server.balancer.TestBalancer
org.apache.hadoop.hdfs.TestDFSClientRetries
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8526//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8526//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8526//console
This message is automatically generated.
> Move checksum computation off the hot path when writing to RAM disk
> ---
>
> Key: HDFS-6934
> URL: https://issues.apache.org/jira/browse/HDFS-6934
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Chris Nauroth
> Attachments: HDFS-6934.3.patch, h6934_20141003b.patch,
> h6934_20141005.patch
>
>
> Since local RAM is considered reliable we can avoid writing checksums on the
> hot path when replicas are being written to a local RAM disk.
> The checksum can be computed by the lazy writer when moving replicas to disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5928) show namespace and namenode ID on NN dfshealth page
[ https://issues.apache.org/jira/browse/HDFS-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183667#comment-14183667 ] Siqi Li commented on HDFS-5928: --- thank you > show namespace and namenode ID on NN dfshealth page > --- > > Key: HDFS-5928 > URL: https://issues.apache.org/jira/browse/HDFS-5928 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Siqi Li >Assignee: Siqi Li > Attachments: HDFS-5928.v2.patch, HDFS-5928.v3.patch, > HDFS-5928.v4.patch, HDFS-5928.v5.patch, HDFS-5928.v6.patch, > HDFS－5928.v1.patch, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7276) Limit the number of byte arrays used by DFSOutputStream
[ https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183664#comment-14183664 ] stack commented on HDFS-7276: - bq. Yes, limiting the number of arrays also limits the number of outstanding packets. Thinking on it, the patch only limits the outstanding number of Packets if Packets always allocate same size buffers, right? If the objective is a limit on Packet count, perhaps a more direct choke on Packet allocations would be more effective? Meantime, I like the buffer reuse that this patch brings. > Limit the number of byte arrays used by DFSOutputStream > --- > > Key: HDFS-7276 > URL: https://issues.apache.org/jira/browse/HDFS-7276 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h7276_20141021.patch, h7276_20141022.patch, > h7276_20141023.patch, h7276_20141024.patch > > > When there are a lot of DFSOutputStream's writing concurrently, the number of > outstanding packets could be large. The byte arrays created by those packets > could occupy a lot of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6934) Move checksum computation off the hot path when writing to RAM disk
[
https://issues.apache.org/jira/browse/HDFS-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183662#comment-14183662
]
Xiaoyu Yao commented on HDFS-6934:
--
Thanks [~cnauroth] for working on this. The fix looks good to me. Below are
some of my comments:
1. FSOutputSummer.java
If FSOutputSummer#getChecksumSize() is added to reduce individual call
sum.getChecksumSize() in FSOutputSummer, there are still three
Sum.getChecksumSize() calls that can be replaced with getChecksumSize().
2. DataChecksum.java
NIT: Can you add spaces between operators of * and + in
DataChecksum#getChecksumSize().
3. BlockReceiver.java
If you put Line 582~583 before Line 574, you can use {code} If
(shouldNotWriteChecksum) {code} instead of duplicating the same logic.
4. DFSInputStream.java
Line 954: This can be moved out of the loop
StorageType[] storageTypes = block.getStorageTypes();
Line 955: Can you add comment on the relationship between index of nodes (i)
to the storage type selection
if (storageTypes != null && i < storageTypes.length) {
storageType = storageTypes[i];
}
5. TestLazyPersistFiles.java
We have a separate test class for SCR related tests in
TestScrLazyPersistFiles.java so that the temporary domain socket does not need
to be created for non-SCR cases. Can put the new SCR cases there?
I notice the test timeout attribute is removed. Is there a particular reason
for that?
DFS_DATANODE_RAM_DISK_LOW_WATERMARK_REPLICAS has been changed to
DFS_DATANODE_RAM_DISK_LOW_WATERMARK_BYTES by HDFS-6988, can you sync with the
trunk and update the patch?
> Move checksum computation off the hot path when writing to RAM disk
> ---
>
> Key: HDFS-6934
> URL: https://issues.apache.org/jira/browse/HDFS-6934
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Chris Nauroth
> Attachments: HDFS-6934.3.patch, h6934_20141003b.patch,
> h6934_20141005.patch
>
>
> Since local RAM is considered reliable we can avoid writing checksums on the
> hot path when replicas are being written to a local RAM disk.
> The checksum can be computed by the lazy writer when moving replicas to disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7235) Can not decommission DN which has invalid block due to bad disk
[
https://issues.apache.org/jira/browse/HDFS-7235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183651#comment-14183651
]
Yongjun Zhang commented on HDFS-7235:
-
Hi Colin,
Good catch of yours. When getLength() throws IOException, the pre-existing
behaviour was to issue a warn message from the caller side of
{{DataNode#transferBlock}}, so I was trying to keep that behaviour. But after
discussing with you, I agree that we should report bad block for this scenario
because the block file was not accessible. I just uploaded rev 006 to
incorporate this change. Thanks.
> Can not decommission DN which has invalid block due to bad disk
> ---
>
> Key: HDFS-7235
> URL: https://issues.apache.org/jira/browse/HDFS-7235
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-7235.001.patch, HDFS-7235.002.patch,
> HDFS-7235.003.patch, HDFS-7235.004.patch, HDFS-7235.005.patch,
> HDFS-7235.006.patch
>
>
> When to decommission a DN, the process hangs.
> What happens is, when NN chooses a replica as a source to replicate data on
> the to-be-decommissioned DN to other DNs, it favors choosing this DN
> to-be-decommissioned as the source of transfer (see BlockManager.java).
> However, because of the bad disk, the DN would detect the source block to be
> transfered as invalidBlock with the following logic in FsDatasetImpl.java:
> {code}
> /** Does the block exist and have the given state? */
> private boolean isValid(final ExtendedBlock b, final ReplicaState state) {
> final ReplicaInfo replicaInfo = volumeMap.get(b.getBlockPoolId(),
> b.getLocalBlock());
> return replicaInfo != null
> && replicaInfo.getState() == state
> && replicaInfo.getBlockFile().exists();
> }
> {code}
> The reason that this method returns false (detecting invalid block) is
> because the block file doesn't exist due to bad disk in this case.
> The key issue we found here is, after DN detects an invalid block for the
> above reason, it doesn't report the invalid block back to NN, thus NN doesn't
> know that the block is corrupted, and keeps sending the data transfer request
> to the same DN to be decommissioned, again and again. This caused an infinite
> loop, so the decommission process hangs.
> Thanks [~qwertymaniac] for reporting the issue and initial analysis.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7235) Can not decommission DN which has invalid block due to bad disk
[
https://issues.apache.org/jira/browse/HDFS-7235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-7235:
Attachment: HDFS-7235.006.patch
> Can not decommission DN which has invalid block due to bad disk
> ---
>
> Key: HDFS-7235
> URL: https://issues.apache.org/jira/browse/HDFS-7235
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-7235.001.patch, HDFS-7235.002.patch,
> HDFS-7235.003.patch, HDFS-7235.004.patch, HDFS-7235.005.patch,
> HDFS-7235.006.patch
>
>
> When to decommission a DN, the process hangs.
> What happens is, when NN chooses a replica as a source to replicate data on
> the to-be-decommissioned DN to other DNs, it favors choosing this DN
> to-be-decommissioned as the source of transfer (see BlockManager.java).
> However, because of the bad disk, the DN would detect the source block to be
> transfered as invalidBlock with the following logic in FsDatasetImpl.java:
> {code}
> /** Does the block exist and have the given state? */
> private boolean isValid(final ExtendedBlock b, final ReplicaState state) {
> final ReplicaInfo replicaInfo = volumeMap.get(b.getBlockPoolId(),
> b.getLocalBlock());
> return replicaInfo != null
> && replicaInfo.getState() == state
> && replicaInfo.getBlockFile().exists();
> }
> {code}
> The reason that this method returns false (detecting invalid block) is
> because the block file doesn't exist due to bad disk in this case.
> The key issue we found here is, after DN detects an invalid block for the
> above reason, it doesn't report the invalid block back to NN, thus NN doesn't
> know that the block is corrupted, and keeps sending the data transfer request
> to the same DN to be decommissioned, again and again. This caused an infinite
> loop, so the decommission process hangs.
> Thanks [~qwertymaniac] for reporting the issue and initial analysis.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7278) Add a command that allows sysadmins to manually trigger full block reports from a DN
[ https://issues.apache.org/jira/browse/HDFS-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183641#comment-14183641 ] Aaron T. Myers commented on HDFS-7278: -- That looks like it'll do it to me. Thanks, Colin. +1 pending Jenkins. > Add a command that allows sysadmins to manually trigger full block reports > from a DN > > > Key: HDFS-7278 > URL: https://issues.apache.org/jira/browse/HDFS-7278 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-7278.002.patch, HDFS-7278.003.patch, > HDFS-7278.004.patch, HDFS-7278.005.patch > > > We should add a command that allows sysadmins to manually trigger full block > reports from a DN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7035) Make adding volume an atomic operation.
[
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183635#comment-14183635
]
Hadoop QA commented on HDFS-7035:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676988/HDFS-7035.007.patch
against trunk revision 501e49f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.TestDFSAdmin
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDistributedFileSystem
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8528//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8528//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8528//console
This message is automatically generated.
> Make adding volume an atomic operation.
> ---
>
> Key: HDFS-7035
> URL: https://issues.apache.org/jira/browse/HDFS-7035
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch,
> HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch,
> HDFS-7035.003.patch, HDFS-7035.003.patch, HDFS-7035.004.patch,
> HDFS-7035.005.patch, HDFS-7035.007.patch, HDFS-7035.008.patch,
> HDFS-7035.009.patch
>
>
> It refactors {{DataStorage}} and {{BlockPoolSliceStorage}} to reduce the
> duplicate code and supports atomic adding volume operations. Also it
> parallels loading data volume operation: each thread loads one volume.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5928) show namespace and namenode ID on NN dfshealth page
[ https://issues.apache.org/jira/browse/HDFS-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183627#comment-14183627 ] Haohui Mai commented on HDFS-5928: -- Looks good to me. +1. I'll commit it shortly. > show namespace and namenode ID on NN dfshealth page > --- > > Key: HDFS-5928 > URL: https://issues.apache.org/jira/browse/HDFS-5928 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Siqi Li >Assignee: Siqi Li > Attachments: HDFS-5928.v2.patch, HDFS-5928.v3.patch, > HDFS-5928.v4.patch, HDFS-5928.v5.patch, HDFS-5928.v6.patch, > HDFS－5928.v1.patch, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7173) Only keep successfully loaded volumes in the configuration.
[
https://issues.apache.org/jira/browse/HDFS-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-7173:
Attachment: HDFS-7173.003.patch
Thanks for the quick reviews, [~cmccabe]. I updated the patch based on your
comments.
> Only keep successfully loaded volumes in the configuration.
> ---
>
> Key: HDFS-7173
> URL: https://issues.apache.org/jira/browse/HDFS-7173
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0, 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7173.000.combo.patch, HDFS-7173.000.patch,
> HDFS-7173.001.combo.patch, HDFS-7173.001.patch, HDFS-7173.002.combo.patch,
> HDFS-7173.002.patch, HDFS-7173.003.combo.patch, HDFS-7173.003.patch
>
>
> Hot swapping data volumes might fail. The user should be able to fix the
> failed volumes and disks, then ask the {{DataNode}} to retry the previously
> failed volumes.
> To attempt to reload the failed volume again on the same directory, this
> failed directory must not be presented in the {{Configuration}} object that
> {{DataNode has}}. Therefore, it should only put successfully loaded volumes
> into the {{Configuration}} object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7173) Only keep successfully loaded volumes in the configuration.
[
https://issues.apache.org/jira/browse/HDFS-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-7173:
Attachment: HDFS-7173.003.combo.patch
> Only keep successfully loaded volumes in the configuration.
> ---
>
> Key: HDFS-7173
> URL: https://issues.apache.org/jira/browse/HDFS-7173
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0, 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7173.000.combo.patch, HDFS-7173.000.patch,
> HDFS-7173.001.combo.patch, HDFS-7173.001.patch, HDFS-7173.002.combo.patch,
> HDFS-7173.002.patch, HDFS-7173.003.combo.patch, HDFS-7173.003.patch
>
>
> Hot swapping data volumes might fail. The user should be able to fix the
> failed volumes and disks, then ask the {{DataNode}} to retry the previously
> failed volumes.
> To attempt to reload the failed volume again on the same directory, this
> failed directory must not be presented in the {{Configuration}} object that
> {{DataNode has}}. Therefore, it should only put successfully loaded volumes
> into the {{Configuration}} object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7276) Limit the number of byte arrays used by DFSOutputStream
[ https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183623#comment-14183623 ] Tsz Wo Nicholas Sze commented on HDFS-7276: --- [~stack], thanks for taking a look. bq. The patch does not seem to be concerned with the amount of memory allocated since it only counts buffers, not how much has been actually allocated. Or is the idea throttle array allocations as means of throttling number of Packets outstanding? Yes, limiting the number of arrays also limits the number of outstanding packets. A full packet is around 64kB. We indeed only limit the outstanding full packets. bq. Any idea how much the synchronize on allocation/recycle and count as well as the wait/notifyAll changes write perf? I will do some performance tests. bq. Why volatile in below when doesn't seem to ever change (Should it be configurable? Would be good if MAX_ARRAYS was configurable at least for clients who'd like to postpone blocking): ... It needs volatile. Otherwise, other threads may not see the change done by reset(..). Configurable is a good idea. Let me think about how to do it. bq. Nit: IMO 1 << 11 is cryptic. Suggest instead you write out 2048? Sure, will do. > Limit the number of byte arrays used by DFSOutputStream > --- > > Key: HDFS-7276 > URL: https://issues.apache.org/jira/browse/HDFS-7276 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h7276_20141021.patch, h7276_20141022.patch, > h7276_20141023.patch, h7276_20141024.patch > > > When there are a lot of DFSOutputStream's writing concurrently, the number of > outstanding packets could be large. The byte arrays created by those packets > could occupy a lot of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6904) YARN unable to renew delegation token fetched via webhdfs due to incorrect service port
[
https://issues.apache.org/jira/browse/HDFS-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDFS-6904:
---
Resolution: Fixed
Fix Version/s: 2.6.0
Status: Resolved (was: Patch Available)
> YARN unable to renew delegation token fetched via webhdfs due to incorrect
> service port
> ---
>
> Key: HDFS-6904
> URL: https://issues.apache.org/jira/browse/HDFS-6904
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Varun Vasudev
>Assignee: Jitendra Nath Pandey
>Priority: Critical
> Fix For: 2.6.0
>
> Attachments: HDFS-6904.1.patch, HDFS-6904.2.patch, HDFS-6904.3.patch,
> HDFS-6904.4.patch
>
>
> YARN is unable to renew delegation tokens obtained via the WebHDFS REST API.
> The scenario is as follows -
> 1. User creates a delegation token using the WebHDFS REST API
> 2. User passes this token to YARN as part of app submission(via the YARN REST
> API)
> 3. When YARN tries to renew this delegation token, it fails because the token
> service is pointing to the RPC port but the token kind is WebHDFS.
> The exception is
> {noformat}
> 2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer
> (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to
> add the application to the delegation token renewer.
> java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation,
> Service: NameNodeIP:8020, Ident: (WEBHDFS delegation token for hrt_qa)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200,
> op=RENEWDELEGATIONTOKEN, message=null
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
> at
> org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
> at org.apache.hadoop.security.token.Token.renew(Token.java:377)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
> ... 6 more
> Caused by: java.io.IOException: The error stream is null.
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:304)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:329)
> ... 24 more
>
[jira] [Commented] (HDFS-7287) The OfflineImageViewer (OIV) can output invalid XML depending on the filename
[
https://issues.apache.org/jira/browse/HDFS-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183608#comment-14183608
]
Hadoop QA commented on HDFS-7287:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676975/HDFS-7287.patch
against trunk revision 86ac0d4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8525//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8525//console
This message is automatically generated.
> The OfflineImageViewer (OIV) can output invalid XML depending on the filename
> -
>
> Key: HDFS-7287
> URL: https://issues.apache.org/jira/browse/HDFS-7287
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HDFS-7287.1.patch, HDFS-7287.patch
>
>
> If the filename contains a character which is invalid in XML,
> TextWriterImageVisitor.write() or PBImageXmlWriter.o() prints out the string
> unescaped. For us this was the character 0x0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7276) Limit the number of byte arrays used by DFSOutputStream
[
https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183607#comment-14183607
]
Hadoop QA commented on HDFS-7276:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676968/h7276_20141024.patch
against trunk revision 86ac0d4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestLeaseRecovery2
org.apache.hadoop.hdfs.server.balancer.TestBalancer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8524//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8524//console
This message is automatically generated.
> Limit the number of byte arrays used by DFSOutputStream
> ---
>
> Key: HDFS-7276
> URL: https://issues.apache.org/jira/browse/HDFS-7276
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Attachments: h7276_20141021.patch, h7276_20141022.patch,
> h7276_20141023.patch, h7276_20141024.patch
>
>
> When there are a lot of DFSOutputStream's writing concurrently, the number of
> outstanding packets could be large. The byte arrays created by those packets
> could occupy a lot of memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5928) show namespace and namenode ID on NN dfshealth page
[ https://issues.apache.org/jira/browse/HDFS-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183602#comment-14183602 ] Siqi Li commented on HDFS-5928: --- [~wheat9] I have updated the patch with a screenshot, the test failures should be irrelevant for this patch > show namespace and namenode ID on NN dfshealth page > --- > > Key: HDFS-5928 > URL: https://issues.apache.org/jira/browse/HDFS-5928 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Siqi Li >Assignee: Siqi Li > Attachments: HDFS-5928.v2.patch, HDFS-5928.v3.patch, > HDFS-5928.v4.patch, HDFS-5928.v5.patch, HDFS-5928.v6.patch, > HDFS－5928.v1.patch, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7173) Only keep successfully loaded volumes in the configuration.
[
https://issues.apache.org/jira/browse/HDFS-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183585#comment-14183585
]
Colin Patrick McCabe commented on HDFS-7173:
{code}
/** The unchanged locations that are existed in the old configuration */
{code}
Should be "the unchanged locations that exist in the old configuration"
There's a whitepsace change in DataNode.java that isn't necessary, can we
remove that?
+1 once those are addressed.
> Only keep successfully loaded volumes in the configuration.
> ---
>
> Key: HDFS-7173
> URL: https://issues.apache.org/jira/browse/HDFS-7173
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0, 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7173.000.combo.patch, HDFS-7173.000.patch,
> HDFS-7173.001.combo.patch, HDFS-7173.001.patch, HDFS-7173.002.combo.patch,
> HDFS-7173.002.patch
>
>
> Hot swapping data volumes might fail. The user should be able to fix the
> failed volumes and disks, then ask the {{DataNode}} to retry the previously
> failed volumes.
> To attempt to reload the failed volume again on the same directory, this
> failed directory must not be presented in the {{Configuration}} object that
> {{DataNode has}}. Therefore, it should only put successfully loaded volumes
> into the {{Configuration}} object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7278) Add a command that allows sysadmins to manually trigger full block reports from a DN
[ https://issues.apache.org/jira/browse/HDFS-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-7278: --- Attachment: HDFS-7278.005.patch [~atm]: Good call. Doing this right is a bit tricky. The latest patch gets it right. I created a synthetic "block deleted" notification which will get sent on the next IBR. But creating this notification doesn't itself trigger an IBR, unlike creating a file. > Add a command that allows sysadmins to manually trigger full block reports > from a DN > > > Key: HDFS-7278 > URL: https://issues.apache.org/jira/browse/HDFS-7278 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-7278.002.patch, HDFS-7278.003.patch, > HDFS-7278.004.patch, HDFS-7278.005.patch > > > We should add a command that allows sysadmins to manually trigger full block > reports from a DN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7173) Only keep successfully loaded volumes in the configuration.
[
https://issues.apache.org/jira/browse/HDFS-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-7173:
Attachment: HDFS-7173.002.patch
[~cmccabe] I changed the 'remainedLocations' to 'unchangedLocations', which I
think is little bit more accurate.
Also the comments are added.
> Only keep successfully loaded volumes in the configuration.
> ---
>
> Key: HDFS-7173
> URL: https://issues.apache.org/jira/browse/HDFS-7173
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0, 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7173.000.combo.patch, HDFS-7173.000.patch,
> HDFS-7173.001.combo.patch, HDFS-7173.001.patch, HDFS-7173.002.combo.patch,
> HDFS-7173.002.patch
>
>
> Hot swapping data volumes might fail. The user should be able to fix the
> failed volumes and disks, then ask the {{DataNode}} to retry the previously
> failed volumes.
> To attempt to reload the failed volume again on the same directory, this
> failed directory must not be presented in the {{Configuration}} object that
> {{DataNode has}}. Therefore, it should only put successfully loaded volumes
> into the {{Configuration}} object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7173) Only keep successfully loaded volumes in the configuration.
[
https://issues.apache.org/jira/browse/HDFS-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-7173:
Attachment: HDFS-7173.002.combo.patch
> Only keep successfully loaded volumes in the configuration.
> ---
>
> Key: HDFS-7173
> URL: https://issues.apache.org/jira/browse/HDFS-7173
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0, 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7173.000.combo.patch, HDFS-7173.000.patch,
> HDFS-7173.001.combo.patch, HDFS-7173.001.patch, HDFS-7173.002.combo.patch,
> HDFS-7173.002.patch
>
>
> Hot swapping data volumes might fail. The user should be able to fix the
> failed volumes and disks, then ask the {{DataNode}} to retry the previously
> failed volumes.
> To attempt to reload the failed volume again on the same directory, this
> failed directory must not be presented in the {{Configuration}} object that
> {{DataNode has}}. Therefore, it should only put successfully loaded volumes
> into the {{Configuration}} object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7287) The OfflineImageViewer (OIV) can output invalid XML depending on the filename
[ https://issues.apache.org/jira/browse/HDFS-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183515#comment-14183515 ] Colin Patrick McCabe commented on HDFS-7287: Thanks, Ravi. +1 from me. I will commit in a day or two if there are no more comments. > The OfflineImageViewer (OIV) can output invalid XML depending on the filename > - > > Key: HDFS-7287 > URL: https://issues.apache.org/jira/browse/HDFS-7287 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Attachments: HDFS-7287.1.patch, HDFS-7287.patch > > > If the filename contains a character which is invalid in XML, > TextWriterImageVisitor.write() or PBImageXmlWriter.o() prints out the string > unescaped. For us this was the character 0x0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7287) The OfflineImageViewer (OIV) can output invalid XML depending on the filename
[ https://issues.apache.org/jira/browse/HDFS-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HDFS-7287: --- Attachment: HDFS-7287.1.patch Thanks Colin for pointing that out! This new patch used XMLUtils. Moreover it just adds 3 seconds to a 10 second fsimage->OIV conversion. > The OfflineImageViewer (OIV) can output invalid XML depending on the filename > - > > Key: HDFS-7287 > URL: https://issues.apache.org/jira/browse/HDFS-7287 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Attachments: HDFS-7287.1.patch, HDFS-7287.patch > > > If the filename contains a character which is invalid in XML, > TextWriterImageVisitor.write() or PBImageXmlWriter.o() prints out the string > unescaped. For us this was the character 0x0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5175) Provide clients a way to set IP header bits on connections
[
https://issues.apache.org/jira/browse/HDFS-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183473#comment-14183473
]
Chris Nauroth commented on HDFS-5175:
-
If there is no net new dependency required, then I'd be +1 for the proposal
overall. It looks like it will be a cleaner way to implement it compared to
trying to deal with the global {{SocketFactory}}. It would be helpful to have
a benchmark showing that there is no performance degradation compared to
{{URLConnection}}.
> Provide clients a way to set IP header bits on connections
> --
>
> Key: HDFS-5175
> URL: https://issues.apache.org/jira/browse/HDFS-5175
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.5-alpha
>Reporter: Lohit Vijayarenu
>
> It would be very helpful if we had ability for clients to set IP headers when
> they make socket connections for data transfers. We were looking into setting
> up QoS using DSCP bit and saw that there is no easy way to let clients pass
> down a specific value when clients make connection to DataNode.
> As a quick fix we did something similar to io.file.buffer.size where client
> could pass down DSCP integer value and when DFSClient opens a stream, it
> could set the value on socket using setTrafficClass
> Opening this JIRA to get more inputs from others who have had experience and
> might have already thought about this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7035) Make adding volume an atomic operation.
[
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-7035:
Attachment: HDFS-7035.009.patch
Update patch to fix conflicts to trunk.
> Make adding volume an atomic operation.
> ---
>
> Key: HDFS-7035
> URL: https://issues.apache.org/jira/browse/HDFS-7035
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch,
> HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch,
> HDFS-7035.003.patch, HDFS-7035.003.patch, HDFS-7035.004.patch,
> HDFS-7035.005.patch, HDFS-7035.007.patch, HDFS-7035.008.patch,
> HDFS-7035.009.patch
>
>
> It refactors {{DataStorage}} and {{BlockPoolSliceStorage}} to reduce the
> duplicate code and supports atomic adding volume operations. Also it
> parallels loading data volume operation: each thread loads one volume.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7287) The OfflineImageViewer (OIV) can output invalid XML depending on the filename
[
https://issues.apache.org/jira/browse/HDFS-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183462#comment-14183462
]
Colin Patrick McCabe commented on HDFS-7287:
Please do not add another dependency for this. We already have a function to
deal with mangling XML:
{{org.apache.hadoop.hdfs.util.XMLUtils#mangleXmlString}}.
> The OfflineImageViewer (OIV) can output invalid XML depending on the filename
> -
>
> Key: HDFS-7287
> URL: https://issues.apache.org/jira/browse/HDFS-7287
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HDFS-7287.patch
>
>
> If the filename contains a character which is invalid in XML,
> TextWriterImageVisitor.write() or PBImageXmlWriter.o() prints out the string
> unescaped. For us this was the character 0x0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7035) Make adding volume an atomic operation.
[
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183454#comment-14183454
]
Hadoop QA commented on HDFS-7035:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12677002/HDFS-7035.008.patch
against trunk revision a52eb4b.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8530//console
This message is automatically generated.
> Make adding volume an atomic operation.
> ---
>
> Key: HDFS-7035
> URL: https://issues.apache.org/jira/browse/HDFS-7035
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch,
> HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch,
> HDFS-7035.003.patch, HDFS-7035.003.patch, HDFS-7035.004.patch,
> HDFS-7035.005.patch, HDFS-7035.007.patch, HDFS-7035.008.patch
>
>
> It refactors {{DataStorage}} and {{BlockPoolSliceStorage}} to reduce the
> duplicate code and supports atomic adding volume operations. Also it
> parallels loading data volume operation: each thread loads one volume.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7258) CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level
[ https://issues.apache.org/jira/browse/HDFS-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183449#comment-14183449 ] Colin Patrick McCabe commented on HDFS-7258: Thanks, [~xyao] and [~wheat9]. > CacheReplicationMonitor rescan schedule log should use DEBUG level instead of > INFO level > > > Key: HDFS-7258 > URL: https://issues.apache.org/jira/browse/HDFS-7258 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Fix For: 2.7.0 > > Attachments: HDFS-7258.0.patch, HDFS-7258.1.patch > > > CacheReplicationMonitor rescan scheduler adds two INFO log entries every 30 > seconds to HDSF NN log as shown below. This should be a DEBUG level log to > avoid flooding the namenode log. > 2014-10-17 07:52:30,265 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 3 milliseconds > 2014-10-17 07:52:30,265 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). > 2014-10-17 07:53:00,265 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 30001 milliseconds > 2014-10-17 07:53:00,266 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s). > 2014-10-17 07:53:30,267 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 30001 milliseconds > 2014-10-17 07:53:30,267 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). > 2014-10-17 07:54:00,267 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 30001 milliseconds > 2014-10-17 07:54:00,268 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). > 2014-10-17 07:54:30,268 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 30001 milliseconds > 2014-10-17 07:54:30,269 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). > 2014-10-17 07:55:00,269 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 3 milliseconds > 2014-10-17 07:55:00,269 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s). > 2014-10-17 07:55:30,268 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 3 milliseconds > 2014-10-17 07:55:30,269 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). > 2014-10-17 07:56:00,269 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 30001 milliseconds > 2014-10-17 07:56:00,270 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). > 2014-10-17 07:56:30,270 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 30001 milliseconds > 2014-10-17 07:56:30,271 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). > 2014-10-17 07:57:00,271 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 3 milliseconds > 2014-10-17 07:57:00,272 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s). > 2014-10-17 07:57:30,271 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 3 milliseconds > 2014-10-17 07:57:30,272 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s). > 2014-10-17 07:58:00,271 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 3 milliseconds > 2014-10-17 07:58:00,271 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s). > 2014-10-17 07:58:30,271 INFO > org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: > Rescanning after 3 milliseconds -- This message was sen
[jira] [Updated] (HDFS-7035) Make adding volume an atomic operation.
[
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-7035:
Attachment: HDFS-7035.008.patch
Update the patch to fix a bug in error message construction.
> Make adding volume an atomic operation.
> ---
>
> Key: HDFS-7035
> URL: https://issues.apache.org/jira/browse/HDFS-7035
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch,
> HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch,
> HDFS-7035.003.patch, HDFS-7035.003.patch, HDFS-7035.004.patch,
> HDFS-7035.005.patch, HDFS-7035.007.patch, HDFS-7035.008.patch
>
>
> It refactors {{DataStorage}} and {{BlockPoolSliceStorage}} to reduce the
> duplicate code and supports atomic adding volume operations. Also it
> parallels loading data volume operation: each thread loads one volume.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7235) Can not decommission DN which has invalid block due to bad disk
[
https://issues.apache.org/jira/browse/HDFS-7235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183444#comment-14183444
]
Colin Patrick McCabe commented on HDFS-7235:
{code}
1791try {
1792 data.checkBlock(block, block.getNumBytes(),
ReplicaState.FINALIZED);
1793} catch (ReplicaNotFoundException e) {
1794 replicaNotExist = true;
1795} catch (UnexpectedReplicaStateException e) {
1796 replicaStateNotFinalized = true;
1797} catch (FileNotFoundException e) {
1798 blockFileNotExist = true;
1799} catch (EOFException e) {
1800 lengthTooShort = true;
1801}
{code}
We should also add a catch block for IOException, which may occur while we're
getting the block file length. Probably this should be handled as
{{blockFileNotExist}}, since the block file was inaccessible.
+1 once this is addressed
> Can not decommission DN which has invalid block due to bad disk
> ---
>
> Key: HDFS-7235
> URL: https://issues.apache.org/jira/browse/HDFS-7235
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-7235.001.patch, HDFS-7235.002.patch,
> HDFS-7235.003.patch, HDFS-7235.004.patch, HDFS-7235.005.patch
>
>
> When to decommission a DN, the process hangs.
> What happens is, when NN chooses a replica as a source to replicate data on
> the to-be-decommissioned DN to other DNs, it favors choosing this DN
> to-be-decommissioned as the source of transfer (see BlockManager.java).
> However, because of the bad disk, the DN would detect the source block to be
> transfered as invalidBlock with the following logic in FsDatasetImpl.java:
> {code}
> /** Does the block exist and have the given state? */
> private boolean isValid(final ExtendedBlock b, final ReplicaState state) {
> final ReplicaInfo replicaInfo = volumeMap.get(b.getBlockPoolId(),
> b.getLocalBlock());
> return replicaInfo != null
> && replicaInfo.getState() == state
> && replicaInfo.getBlockFile().exists();
> }
> {code}
> The reason that this method returns false (detecting invalid block) is
> because the block file doesn't exist due to bad disk in this case.
> The key issue we found here is, after DN detects an invalid block for the
> above reason, it doesn't report the invalid block back to NN, thus NN doesn't
> know that the block is corrupted, and keeps sending the data transfer request
> to the same DN to be decommissioned, again and again. This caused an infinite
> loop, so the decommission process hangs.
> Thanks [~qwertymaniac] for reporting the issue and initial analysis.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7287) The OfflineImageViewer (OIV) can output invalid XML depending on the filename
[
https://issues.apache.org/jira/browse/HDFS-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183422#comment-14183422
]
Ravi Prakash commented on HDFS-7287:
Hi Haohui! I'm not familiar with CDATA, but the wikipedia
https://en.wikipedia.org/wiki/CDATA entry says
{quote} But the text within a CDATA section is strictly limited to the
characters available in the encoding.
Because of this, using a CDATA section programmatically to quote data that
could potentially contain '&' or '<' characters can cause problems when the
data happens to contain characters that cannot be represented in the encoding
{quote}
> The OfflineImageViewer (OIV) can output invalid XML depending on the filename
> -
>
> Key: HDFS-7287
> URL: https://issues.apache.org/jira/browse/HDFS-7287
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HDFS-7287.patch
>
>
> If the filename contains a character which is invalid in XML,
> TextWriterImageVisitor.write() or PBImageXmlWriter.o() prints out the string
> unescaped. For us this was the character 0x0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6988) Improve HDFS-6581 eviction configuration
[ https://issues.apache.org/jira/browse/HDFS-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183417#comment-14183417 ] Xiaoyu Yao commented on HDFS-6988: -- Thanks [~cmccabe] again for reviewing and committing the patch. > Improve HDFS-6581 eviction configuration > > > Key: HDFS-6988 > URL: https://issues.apache.org/jira/browse/HDFS-6988 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: 2.6.0 >Reporter: Arpit Agarwal >Assignee: Xiaoyu Yao > Fix For: 2.6.0 > > Attachments: HDFS-6988.01.patch, HDFS-6988.02.patch, > HDFS-6988.03.patch, HDFS-6988.4.patch > > > Per feedback from [~cmccabe] on HDFS-6930, we can make the eviction > thresholds configurable. The hard-coded thresholds may not be appropriate for > very large RAM disks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7286) Remove dead code in ServletUtil
[ https://issues.apache.org/jira/browse/HDFS-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183415#comment-14183415 ] Haohui Mai commented on HDFS-7286: -- The test failure is unrelated. +1. I'll commit it shortly. > Remove dead code in ServletUtil > --- > > Key: HDFS-7286 > URL: https://issues.apache.org/jira/browse/HDFS-7286 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Li Lu >Priority: Minor > Attachments: HDFS-7286-102414.patch > > > Some code in ServletUtil is dead after the JSP UI is phased out. This jira > propose to clean them up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7286) Remove dead code in ServletUtil
[
https://issues.apache.org/jira/browse/HDFS-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183409#comment-14183409
]
Hadoop QA commented on HDFS-7286:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676979/HDFS-7286-102414.patch
against trunk revision 86ac0d4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8527//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8527//console
This message is automatically generated.
> Remove dead code in ServletUtil
> ---
>
> Key: HDFS-7286
> URL: https://issues.apache.org/jira/browse/HDFS-7286
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Li Lu
>Priority: Minor
> Attachments: HDFS-7286-102414.patch
>
>
> Some code in ServletUtil is dead after the JSP UI is phased out. This jira
> propose to clean them up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6988) Improve HDFS-6581 eviction configuration
[ https://issues.apache.org/jira/browse/HDFS-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-6988: --- Resolution: Fixed Fix Version/s: (was: 3.0.0) 2.6.0 Target Version/s: 2.6.0 Status: Resolved (was: Patch Available) Committed to 2.6. Thanks, Xiaoyu. > Improve HDFS-6581 eviction configuration > > > Key: HDFS-6988 > URL: https://issues.apache.org/jira/browse/HDFS-6988 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: 2.6.0 >Reporter: Arpit Agarwal >Assignee: Xiaoyu Yao > Fix For: 2.6.0 > > Attachments: HDFS-6988.01.patch, HDFS-6988.02.patch, > HDFS-6988.03.patch, HDFS-6988.4.patch > > > Per feedback from [~cmccabe] on HDFS-6930, we can make the eviction > thresholds configurable. The hard-coded thresholds may not be appropriate for > very large RAM disks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7278) Add a command that allows sysadmins to manually trigger full block reports from a DN
[ https://issues.apache.org/jira/browse/HDFS-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-7278: --- Attachment: HDFS-7278.004.patch Add escaping to "reconfig" part of .apt.vm > Add a command that allows sysadmins to manually trigger full block reports > from a DN > > > Key: HDFS-7278 > URL: https://issues.apache.org/jira/browse/HDFS-7278 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-7278.002.patch, HDFS-7278.003.patch, > HDFS-7278.004.patch > > > We should add a command that allows sysadmins to manually trigger full block > reports from a DN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6988) Improve HDFS-6581 eviction configuration
[ https://issues.apache.org/jira/browse/HDFS-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183403#comment-14183403 ] Hudson commented on HDFS-6988: -- FAILURE: Integrated in Hadoop-trunk-Commit #6338 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6338/]) HDFS-6988. Improve HDFS-6581 eviction configuration (Xiaoyu Yao via Colin P. McCabe) (cmccabe: rev a52eb4bc5fb21574859f779001ea9d95bf5207fe) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java > Improve HDFS-6581 eviction configuration > > > Key: HDFS-6988 > URL: https://issues.apache.org/jira/browse/HDFS-6988 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: 2.6.0 >Reporter: Arpit Agarwal >Assignee: Xiaoyu Yao > Fix For: 3.0.0 > > Attachments: HDFS-6988.01.patch, HDFS-6988.02.patch, > HDFS-6988.03.patch, HDFS-6988.4.patch > > > Per feedback from [~cmccabe] on HDFS-6930, we can make the eviction > thresholds configurable. The hard-coded thresholds may not be appropriate for > very large RAM disks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6581) Write to single replica in memory
[ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183404#comment-14183404 ] Hudson commented on HDFS-6581: -- FAILURE: Integrated in Hadoop-trunk-Commit #6338 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6338/]) HDFS-6988. Improve HDFS-6581 eviction configuration (Xiaoyu Yao via Colin P. McCabe) (cmccabe: rev a52eb4bc5fb21574859f779001ea9d95bf5207fe) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java > Write to single replica in memory > - > > Key: HDFS-6581 > URL: https://issues.apache.org/jira/browse/HDFS-6581 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.6.0 > > Attachments: HDFS-6581.merge.01.patch, HDFS-6581.merge.02.patch, > HDFS-6581.merge.03.patch, HDFS-6581.merge.04.patch, HDFS-6581.merge.05.patch, > HDFS-6581.merge.06.patch, HDFS-6581.merge.07.patch, HDFS-6581.merge.08.patch, > HDFS-6581.merge.09.patch, HDFS-6581.merge.10.patch, HDFS-6581.merge.11.patch, > HDFS-6581.merge.12.patch, HDFS-6581.merge.14.patch, HDFS-6581.merge.15.patch, > HDFSWriteableReplicasInMemory.pdf, > Test-Plan-for-HDFS-6581-Memory-Storage.pdf, > Test-Plan-for-HDFS-6581-Memory-Storage.pdf > > > Per discussion with the community on HDFS-5851, we will implement writing to > a single replica in DN memory via DataTransferProtocol. > This avoids some of the issues with short-circuit writes, which we can > revisit at a later time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7278) Add a command that allows sysadmins to manually trigger full block reports from a DN
[ https://issues.apache.org/jira/browse/HDFS-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183401#comment-14183401 ] Aaron T. Myers commented on HDFS-7278: -- Latest patch looks pretty good to me, though I think I've noticed another race condition in the test. In the case of testing the incremental IBR trigger, I believe that the IBR will be sent promptly regardless of whether or not the heartbeat is set to a long time or the triggerBlockReport command is sent, which means that the test can pass without the admin command actually being run. To test this out, I commented out the call to triggerBlockReport, and indeed that test still passes, although the full BR test correctly fails. +1 once this and Akira's doc comment are addressed. > Add a command that allows sysadmins to manually trigger full block reports > from a DN > > > Key: HDFS-7278 > URL: https://issues.apache.org/jira/browse/HDFS-7278 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-7278.002.patch, HDFS-7278.003.patch > > > We should add a command that allows sysadmins to manually trigger full block > reports from a DN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7276) Limit the number of byte arrays used by DFSOutputStream
[ https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183398#comment-14183398 ] stack commented on HDFS-7276: - [~szetszwo] bq. ...the number of outstanding packets could be large. The byte arrays created by those packets could occupy a lot of memory. The patch does not seem to be concerned with the amount of memory allocated since it only counts buffers, not how much has been actually allocated. Or is the idea throttle array allocations as means of throttling number of Packets outstanding? On quick review of the patch, this patch is about (or also about) reusing buffers (nice). Any idea how much the synchronize on allocation/recycle and count as well as the wait/notifyAll changes write perf? Suggest ByteArrayManager gets a class comment on what it does especially as a bunch going on in here. Why volatile in below when doesn't seem to ever change (Should it be configurable? Would be good if MAX_ARRAYS was configurable at least for clients who'd like to postpone blocking): private static volatile long COUNT_RESET_TIME_PERIOD_MS = 10L * 1000; Nit: IMO 1 << 11 is cryptic. Suggest instead you write out 2048? Good stuff. > Limit the number of byte arrays used by DFSOutputStream > --- > > Key: HDFS-7276 > URL: https://issues.apache.org/jira/browse/HDFS-7276 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h7276_20141021.patch, h7276_20141022.patch, > h7276_20141023.patch, h7276_20141024.patch > > > When there are a lot of DFSOutputStream's writing concurrently, the number of > outstanding packets could be large. The byte arrays created by those packets > could occupy a lot of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6988) Improve HDFS-6581 eviction configuration
[ https://issues.apache.org/jira/browse/HDFS-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-6988: --- Summary: Improve HDFS-6581 eviction configuration (was: Add configurable limit for percentage-based eviction threshold) > Improve HDFS-6581 eviction configuration > > > Key: HDFS-6988 > URL: https://issues.apache.org/jira/browse/HDFS-6988 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: 2.6.0 >Reporter: Arpit Agarwal >Assignee: Xiaoyu Yao > Fix For: 3.0.0 > > Attachments: HDFS-6988.01.patch, HDFS-6988.02.patch, > HDFS-6988.03.patch, HDFS-6988.4.patch > > > Per feedback from [~cmccabe] on HDFS-6930, we can make the eviction > thresholds configurable. The hard-coded thresholds may not be appropriate for > very large RAM disks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7287) The OfflineImageViewer (OIV) can output invalid XML depending on the filename
[
https://issues.apache.org/jira/browse/HDFS-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183386#comment-14183386
]
Haohui Mai commented on HDFS-7287:
--
As a first cut, is it possible to wrap the data with the {{CDATA}} tag?
> The OfflineImageViewer (OIV) can output invalid XML depending on the filename
> -
>
> Key: HDFS-7287
> URL: https://issues.apache.org/jira/browse/HDFS-7287
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HDFS-7287.patch
>
>
> If the filename contains a character which is invalid in XML,
> TextWriterImageVisitor.write() or PBImageXmlWriter.o() prints out the string
> unescaped. For us this was the character 0x0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5175) Provide clients a way to set IP header bits on connections
[ https://issues.apache.org/jira/browse/HDFS-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183373#comment-14183373 ] Ming Ma commented on HDFS-5175: --- Yeah, we do have dependency on org.apache.httpcomponents' httpclient. That might be enough for HDFS. My main question is if folks are ok to have webHDFS use org.apache.httpcomponents' httpclient instead of URLConnection. > Provide clients a way to set IP header bits on connections > -- > > Key: HDFS-5175 > URL: https://issues.apache.org/jira/browse/HDFS-5175 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.5-alpha >Reporter: Lohit Vijayarenu > > It would be very helpful if we had ability for clients to set IP headers when > they make socket connections for data transfers. We were looking into setting > up QoS using DSCP bit and saw that there is no easy way to let clients pass > down a specific value when clients make connection to DataNode. > As a quick fix we did something similar to io.file.buffer.size where client > could pass down DSCP integer value and when DFSClient opens a stream, it > could set the value on socket using setTrafficClass > Opening this JIRA to get more inputs from others who have had experience and > might have already thought about this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7035) Make adding volume an atomic operation.
[
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-7035:
Attachment: HDFS-7035.007.patch
Hi, [~cmccabe] Thanks for the reviews. I have removed the new class
hierarchical in this patch.
I clear the IOException catching code, there is no other Exception, so I
removed the "throws Exception" from the {{call()}} signature.
Could you take another look?
> Make adding volume an atomic operation.
> ---
>
> Key: HDFS-7035
> URL: https://issues.apache.org/jira/browse/HDFS-7035
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch,
> HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch,
> HDFS-7035.003.patch, HDFS-7035.003.patch, HDFS-7035.004.patch,
> HDFS-7035.005.patch, HDFS-7035.007.patch
>
>
> It refactors {{DataStorage}} and {{BlockPoolSliceStorage}} to reduce the
> duplicate code and supports atomic adding volume operations. Also it
> parallels loading data volume operation: each thread loads one volume.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6934) Move checksum computation off the hot path when writing to RAM disk
[
https://issues.apache.org/jira/browse/HDFS-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-6934:
Attachment: HDFS-6934.3.patch
I'm resuming the work started on this by Nicholas. I'm uploading patch v3. I
believe this addresses the prior feedback from Jing. Here is a summary of the
changes in the current patch.
{{FSOutputSummer}}, {{Options}}, {{DataChecksum}}: Refactoring to help simplify
usage of the checksum classes in HDFS.
{{BlockReaderFactory}}: Pass along storage type for use later by individual
block reader classes.
{{BlockReaderLocal}}: Allow skipping checksums if the replica is on transient
storage. However, do not anchor, because the intent is not that clients should
prevent the DataNode from evicting a replica from RAM.
{{BlockReaderLocalLegacy}}: Skip checksums when reading a replica on transient
storage. Do not cache path information for a replica on transient storage,
because eviction at the DataNode may later invalidate that cached path. Unlike
the newer short-circuit read implementation, we don't have a communication
channel (the shared memory segment) for the DataNode to communicate path
invalidation back to the client, so we don't have a more graceful way to handle
this.
{{DFSClient}}: Minor changes associated with refactoring of the checksum
classes in Common.
{{DFSInputStream}}: Get storage type from the {{LocatedBlock}}. Pass it along
to {{BlockReaderFactory}}.
{{DFSOutputStream}}: Do not write checksums if using transient storage. Minor
changes associated with refactoring of the checksum classes in Common.
{{LocatedBlock}}: Change {{toString}} to aid debugging.
{{BlockMetadataHeader}}: Similar to the changes in Common, this is refactoring
to make code related to checksum handling simpler.
{{BlockReceiver}}: Do not require writing checksums on transient storage.
{{BlockSender}}: Do not verify checksum when reading a replica from transient
storage.
{{ReplicaInPipeline}}: Minor cleanups.
{{ReplicaOutputStreams}}: Expose whether or not the replica is on transient
storage.
{{BlockPoolSlice}}: Minor change to go along with the refactoring of checksum
classes.
{{FsDatasetImpl}}: During lazy persist, instead of copying a meta file from
transient storage, we now calculate the checksum to create a new meta file.
Also, when a replica gets evicted from transient storage, make a call to the
{{ShortCircuitRegistry}} to inform clients that they must evict from their
caches.
{{RamDiskAsyncLazyPersistService}}: Log stack trace for lazy persist failures
to aid debugging.
{{RamDiskReplicaLruTracker}}: Minor clean-up of raw generic type.
{{TestLazyPersistFiles}}: New tests added covering short-circuit read (new and
legacy) in combination with eviction, including corruption detection.
Refactored to use a global timeout setting.
> Move checksum computation off the hot path when writing to RAM disk
> ---
>
> Key: HDFS-6934
> URL: https://issues.apache.org/jira/browse/HDFS-6934
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Tsz Wo Nicholas Sze
> Attachments: HDFS-6934.3.patch, h6934_20141003b.patch,
> h6934_20141005.patch
>
>
> Since local RAM is considered reliable we can avoid writing checksums on the
> hot path when replicas are being written to a local RAM disk.
> The checksum can be computed by the lazy writer when moving replicas to disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-6934) Move checksum computation off the hot path when writing to RAM disk
[ https://issues.apache.org/jira/browse/HDFS-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth reassigned HDFS-6934: --- Assignee: Chris Nauroth (was: Tsz Wo Nicholas Sze) > Move checksum computation off the hot path when writing to RAM disk > --- > > Key: HDFS-6934 > URL: https://issues.apache.org/jira/browse/HDFS-6934 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Arpit Agarwal >Assignee: Chris Nauroth > Attachments: HDFS-6934.3.patch, h6934_20141003b.patch, > h6934_20141005.patch > > > Since local RAM is considered reliable we can avoid writing checksums on the > hot path when replicas are being written to a local RAM disk. > The checksum can be computed by the lazy writer when moving replicas to disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7286) Remove dead code in ServletUtil
[ https://issues.apache.org/jira/browse/HDFS-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu updated HDFS-7286: Status: Patch Available (was: Open) > Remove dead code in ServletUtil > --- > > Key: HDFS-7286 > URL: https://issues.apache.org/jira/browse/HDFS-7286 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Li Lu >Priority: Minor > Attachments: HDFS-7286-102414.patch > > > Some code in ServletUtil is dead after the JSP UI is phased out. This jira > propose to clean them up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7286) Remove dead code in ServletUtil
[ https://issues.apache.org/jira/browse/HDFS-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu updated HDFS-7286: Attachment: HDFS-7286-102414.patch Hi [~wheat9], I've removed the dead code in ServletUtil. Would you please take a quick look at the patch? Thanks! > Remove dead code in ServletUtil > --- > > Key: HDFS-7286 > URL: https://issues.apache.org/jira/browse/HDFS-7286 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Li Lu >Priority: Minor > Attachments: HDFS-7286-102414.patch > > > Some code in ServletUtil is dead after the JSP UI is phased out. This jira > propose to clean them up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6904) YARN unable to renew delegation token fetched via webhdfs due to incorrect service port
[
https://issues.apache.org/jira/browse/HDFS-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183322#comment-14183322
]
Hudson commented on HDFS-6904:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #6336 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6336/])
HDFS-6904. Added files missing in previous commit. (jitendra: rev
86ac0d40b850069e9dc6a7930b75a45902803357)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/TokenKindParam.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/TokenServiceParam.java
> YARN unable to renew delegation token fetched via webhdfs due to incorrect
> service port
> ---
>
> Key: HDFS-6904
> URL: https://issues.apache.org/jira/browse/HDFS-6904
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Varun Vasudev
>Assignee: Jitendra Nath Pandey
>Priority: Critical
> Attachments: HDFS-6904.1.patch, HDFS-6904.2.patch, HDFS-6904.3.patch,
> HDFS-6904.4.patch
>
>
> YARN is unable to renew delegation tokens obtained via the WebHDFS REST API.
> The scenario is as follows -
> 1. User creates a delegation token using the WebHDFS REST API
> 2. User passes this token to YARN as part of app submission(via the YARN REST
> API)
> 3. When YARN tries to renew this delegation token, it fails because the token
> service is pointing to the RPC port but the token kind is WebHDFS.
> The exception is
> {noformat}
> 2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer
> (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to
> add the application to the delegation token renewer.
> java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation,
> Service: NameNodeIP:8020, Ident: (WEBHDFS delegation token for hrt_qa)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200,
> op=RENEWDELEGATIONTOKEN, message=null
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
> at
> org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
> at org.apache.hadoop.security.token.Token.renew(Token.java:377)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(Delega
[jira] [Resolved] (HDFS-7288) HDFS compiling failure on trunk
[
https://issues.apache.org/jira/browse/HDFS-7288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-7288.
-
Resolution: Not a Problem
Fix Version/s: 2.6.0
Thanks to [~jnp] for fixing this so quicky in HDFS-6904. I'm resolving this as
duplicate.
> HDFS compiling failure on trunk
> ---
>
> Key: HDFS-7288
> URL: https://issues.apache.org/jira/browse/HDFS-7288
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhijie Shen
>Priority: Blocker
> Fix For: 2.6.0
>
>
> See
> https://builds.apache.org/job/PreCommit-YARN-Build/5545/artifact/patchprocess/trunkJavacWarnings.txt
> {code}
> [INFO] 40 errors
> [INFO] -
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Hadoop Main SUCCESS [ 0.285 s]
> [INFO] Apache Hadoop Project POM . SUCCESS [ 0.462 s]
> [INFO] Apache Hadoop Annotations . SUCCESS [ 1.368 s]
> [INFO] Apache Hadoop Project Dist POM SUCCESS [ 0.116 s]
> [INFO] Apache Hadoop Assemblies .. SUCCESS [ 0.108 s]
> [INFO] Apache Hadoop Maven Plugins ... SUCCESS [ 2.382 s]
> [INFO] Apache Hadoop MiniKDC . SUCCESS [ 2.612 s]
> [INFO] Apache Hadoop Auth SUCCESS [ 3.365 s]
> [INFO] Apache Hadoop Auth Examples ... SUCCESS [ 1.018 s]
> [INFO] Apache Hadoop Common .. SUCCESS [ 20.312 s]
> [INFO] Apache Hadoop NFS . SUCCESS [ 3.382 s]
> [INFO] Apache Hadoop KMS . SUCCESS [ 4.705 s]
> [INFO] Apache Hadoop Common Project .. SUCCESS [ 0.029 s]
> [INFO] Apache Hadoop HDFS FAILURE [ 25.008 s]
> [INFO] Apache Hadoop HttpFS .. SKIPPED
> [INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
> [INFO] Apache Hadoop HDFS-NFS SKIPPED
> [INFO] Apache Hadoop HDFS Project SKIPPED
> [INFO] hadoop-yarn ... SKIPPED
> [INFO] hadoop-yarn-api ... SKIPPED
> [INFO] hadoop-yarn-common SKIPPED
> [INFO] hadoop-yarn-server SKIPPED
> [INFO] hadoop-yarn-server-common . SKIPPED
> [INFO] hadoop-yarn-server-nodemanager SKIPPED
> [INFO] hadoop-yarn-server-web-proxy .. SKIPPED
> [INFO] hadoop-yarn-server-applicationhistoryservice .. SKIPPED
> [INFO] hadoop-yarn-server-resourcemanager SKIPPED
> [INFO] hadoop-yarn-server-tests .. SKIPPED
> [INFO] hadoop-yarn-client SKIPPED
> [INFO] hadoop-yarn-server-sharedcachemanager . SKIPPED
> [INFO] hadoop-yarn-applications .. SKIPPED
> [INFO] hadoop-yarn-applications-distributedshell . SKIPPED
> [INFO] hadoop-yarn-applications-unmanaged-am-launcher SKIPPED
> [INFO] hadoop-yarn-site .. SKIPPED
> [INFO] hadoop-yarn-registry .. SKIPPED
> [INFO] hadoop-yarn-project ... SKIPPED
> [INFO] hadoop-mapreduce-client ... SKIPPED
> [INFO] hadoop-mapreduce-client-core .. SKIPPED
> [INFO] hadoop-mapreduce-client-common SKIPPED
> [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED
> [INFO] hadoop-mapreduce-client-app ... SKIPPED
> [INFO] hadoop-mapreduce-client-hs SKIPPED
> [INFO] hadoop-mapreduce-client-jobclient . SKIPPED
> [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED
> [INFO] hadoop-mapreduce-client-nativetask SKIPPED
> [INFO] Apache Hadoop MapReduce Examples .. SKIPPED
> [INFO] hadoop-mapreduce .. SKIPPED
> [INFO] Apache Hadoop MapReduce Streaming . SKIPPED
> [INFO] Apache Hadoop Distributed Copy SKIPPED
> [INFO] Apache Hadoop Archives SKIPPED
> [INFO] Apache Hadoop Rumen ... SKIPPED
> [INFO] Apache Hadoop Gridmix . SKIPPED
> [INFO] Apache Hadoop Data Join ... SKIPPED
> [INFO] Apache Hadoop Ant Tasks ... SKIPPED
> [INFO] Apache Hadoop Extras .. SKIPPED
> [INFO] Apache Hadoop
[jira] [Commented] (HDFS-6741) Improve permission denied message when FSPermissionChecker#checkOwner fails
[
https://issues.apache.org/jira/browse/HDFS-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183315#comment-14183315
]
Hadoop QA commented on HDFS-6741:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12657520/HDFS-6741.1.patch
against trunk revision 0942c99.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestRollingUpgrade
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8522//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8522//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8522//console
This message is automatically generated.
> Improve permission denied message when FSPermissionChecker#checkOwner fails
> ---
>
> Key: HDFS-6741
> URL: https://issues.apache.org/jira/browse/HDFS-6741
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.5.0
>Reporter: Stephen Chu
>Assignee: Stephen Chu
>Priority: Trivial
> Labels: supportability
> Attachments: HDFS-6741.1.patch
>
>
> Currently, FSPermissionChecker#checkOwner throws an AccessControlException
> with a simple "Permission denied" message.
> When users try to set an ACL without ownership permissions, they'll see
> something like:
> {code}
> [schu@hdfs-vanilla-1 hadoop]$ hdfs dfs -setfacl -m user:schu:--- /tmp
> setfacl: Permission denied
> {code}
> It'd be helpful if the message had an explanation why the permission was
> denied to avoid confusion for users who aren't familiar with permissions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7287) The OfflineImageViewer (OIV) can output invalid XML depending on the filename
[ https://issues.apache.org/jira/browse/HDFS-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HDFS-7287: --- Assignee: Ravi Prakash Status: Patch Available (was: Open) > The OfflineImageViewer (OIV) can output invalid XML depending on the filename > - > > Key: HDFS-7287 > URL: https://issues.apache.org/jira/browse/HDFS-7287 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Attachments: HDFS-7287.patch > > > If the filename contains a character which is invalid in XML, > TextWriterImageVisitor.write() or PBImageXmlWriter.o() prints out the string > unescaped. For us this was the character 0x0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7287) The OfflineImageViewer (OIV) can output invalid XML depending on the filename
[ https://issues.apache.org/jira/browse/HDFS-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HDFS-7287: --- Attachment: HDFS-7287.patch Words of warning: 1. I am *adding* apache.commons.lang3 to the list of dependencies for hadoop-hdfs 2. StringEscapeUtils.escapeXml10 is slowing down OIV considerably. A file which used to take 10 seconds is now taking 50 seconds. LANG-877 talks about performance improvements for StringUtils, but to me this is a correctness issue > The OfflineImageViewer (OIV) can output invalid XML depending on the filename > - > > Key: HDFS-7287 > URL: https://issues.apache.org/jira/browse/HDFS-7287 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Ravi Prakash > Attachments: HDFS-7287.patch > > > If the filename contains a character which is invalid in XML, > TextWriterImageVisitor.write() or PBImageXmlWriter.o() prints out the string > unescaped. For us this was the character 0x0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7288) HDFS compiling failure on trunk
[
https://issues.apache.org/jira/browse/HDFS-7288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-7288:
Priority: Blocker (was: Major)
Target Version/s: 2.6.0
> HDFS compiling failure on trunk
> ---
>
> Key: HDFS-7288
> URL: https://issues.apache.org/jira/browse/HDFS-7288
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhijie Shen
>Priority: Blocker
>
> See
> https://builds.apache.org/job/PreCommit-YARN-Build/5545/artifact/patchprocess/trunkJavacWarnings.txt
> {code}
> [INFO] 40 errors
> [INFO] -
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Hadoop Main SUCCESS [ 0.285 s]
> [INFO] Apache Hadoop Project POM . SUCCESS [ 0.462 s]
> [INFO] Apache Hadoop Annotations . SUCCESS [ 1.368 s]
> [INFO] Apache Hadoop Project Dist POM SUCCESS [ 0.116 s]
> [INFO] Apache Hadoop Assemblies .. SUCCESS [ 0.108 s]
> [INFO] Apache Hadoop Maven Plugins ... SUCCESS [ 2.382 s]
> [INFO] Apache Hadoop MiniKDC . SUCCESS [ 2.612 s]
> [INFO] Apache Hadoop Auth SUCCESS [ 3.365 s]
> [INFO] Apache Hadoop Auth Examples ... SUCCESS [ 1.018 s]
> [INFO] Apache Hadoop Common .. SUCCESS [ 20.312 s]
> [INFO] Apache Hadoop NFS . SUCCESS [ 3.382 s]
> [INFO] Apache Hadoop KMS . SUCCESS [ 4.705 s]
> [INFO] Apache Hadoop Common Project .. SUCCESS [ 0.029 s]
> [INFO] Apache Hadoop HDFS FAILURE [ 25.008 s]
> [INFO] Apache Hadoop HttpFS .. SKIPPED
> [INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
> [INFO] Apache Hadoop HDFS-NFS SKIPPED
> [INFO] Apache Hadoop HDFS Project SKIPPED
> [INFO] hadoop-yarn ... SKIPPED
> [INFO] hadoop-yarn-api ... SKIPPED
> [INFO] hadoop-yarn-common SKIPPED
> [INFO] hadoop-yarn-server SKIPPED
> [INFO] hadoop-yarn-server-common . SKIPPED
> [INFO] hadoop-yarn-server-nodemanager SKIPPED
> [INFO] hadoop-yarn-server-web-proxy .. SKIPPED
> [INFO] hadoop-yarn-server-applicationhistoryservice .. SKIPPED
> [INFO] hadoop-yarn-server-resourcemanager SKIPPED
> [INFO] hadoop-yarn-server-tests .. SKIPPED
> [INFO] hadoop-yarn-client SKIPPED
> [INFO] hadoop-yarn-server-sharedcachemanager . SKIPPED
> [INFO] hadoop-yarn-applications .. SKIPPED
> [INFO] hadoop-yarn-applications-distributedshell . SKIPPED
> [INFO] hadoop-yarn-applications-unmanaged-am-launcher SKIPPED
> [INFO] hadoop-yarn-site .. SKIPPED
> [INFO] hadoop-yarn-registry .. SKIPPED
> [INFO] hadoop-yarn-project ... SKIPPED
> [INFO] hadoop-mapreduce-client ... SKIPPED
> [INFO] hadoop-mapreduce-client-core .. SKIPPED
> [INFO] hadoop-mapreduce-client-common SKIPPED
> [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED
> [INFO] hadoop-mapreduce-client-app ... SKIPPED
> [INFO] hadoop-mapreduce-client-hs SKIPPED
> [INFO] hadoop-mapreduce-client-jobclient . SKIPPED
> [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED
> [INFO] hadoop-mapreduce-client-nativetask SKIPPED
> [INFO] Apache Hadoop MapReduce Examples .. SKIPPED
> [INFO] hadoop-mapreduce .. SKIPPED
> [INFO] Apache Hadoop MapReduce Streaming . SKIPPED
> [INFO] Apache Hadoop Distributed Copy SKIPPED
> [INFO] Apache Hadoop Archives SKIPPED
> [INFO] Apache Hadoop Rumen ... SKIPPED
> [INFO] Apache Hadoop Gridmix . SKIPPED
> [INFO] Apache Hadoop Data Join ... SKIPPED
> [INFO] Apache Hadoop Ant Tasks ... SKIPPED
> [INFO] Apache Hadoop Extras .. SKIPPED
> [INFO] Apache Hadoop Pipes ... SKIPPED
> [INFO] Apache Hadoop OpenStack support ... SKIPP
[jira] [Commented] (HDFS-6904) YARN unable to renew delegation token fetched via webhdfs due to incorrect service port
[
https://issues.apache.org/jira/browse/HDFS-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183296#comment-14183296
]
Jitendra Nath Pandey commented on HDFS-6904:
Thanks Chris. Committed the missing files!
> YARN unable to renew delegation token fetched via webhdfs due to incorrect
> service port
> ---
>
> Key: HDFS-6904
> URL: https://issues.apache.org/jira/browse/HDFS-6904
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Varun Vasudev
>Assignee: Jitendra Nath Pandey
>Priority: Critical
> Attachments: HDFS-6904.1.patch, HDFS-6904.2.patch, HDFS-6904.3.patch,
> HDFS-6904.4.patch
>
>
> YARN is unable to renew delegation tokens obtained via the WebHDFS REST API.
> The scenario is as follows -
> 1. User creates a delegation token using the WebHDFS REST API
> 2. User passes this token to YARN as part of app submission(via the YARN REST
> API)
> 3. When YARN tries to renew this delegation token, it fails because the token
> service is pointing to the RPC port but the token kind is WebHDFS.
> The exception is
> {noformat}
> 2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer
> (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to
> add the application to the delegation token renewer.
> java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation,
> Service: NameNodeIP:8020, Ident: (WEBHDFS delegation token for hrt_qa)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200,
> op=RENEWDELEGATIONTOKEN, message=null
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
> at
> org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
> at org.apache.hadoop.security.token.Token.renew(Token.java:377)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
> ... 6 more
> Caused by: java.io.IOException: The error stream is null.
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:304)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:329)
> ... 24 more
> 2014-08-19 03:12:54,735 DEBUG e
[jira] [Commented] (HDFS-6904) YARN unable to renew delegation token fetched via webhdfs due to incorrect service port
[
https://issues.apache.org/jira/browse/HDFS-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183288#comment-14183288
]
Chris Nauroth commented on HDFS-6904:
-
Thank you!
> YARN unable to renew delegation token fetched via webhdfs due to incorrect
> service port
> ---
>
> Key: HDFS-6904
> URL: https://issues.apache.org/jira/browse/HDFS-6904
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Varun Vasudev
>Assignee: Jitendra Nath Pandey
>Priority: Critical
> Attachments: HDFS-6904.1.patch, HDFS-6904.2.patch, HDFS-6904.3.patch,
> HDFS-6904.4.patch
>
>
> YARN is unable to renew delegation tokens obtained via the WebHDFS REST API.
> The scenario is as follows -
> 1. User creates a delegation token using the WebHDFS REST API
> 2. User passes this token to YARN as part of app submission(via the YARN REST
> API)
> 3. When YARN tries to renew this delegation token, it fails because the token
> service is pointing to the RPC port but the token kind is WebHDFS.
> The exception is
> {noformat}
> 2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer
> (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to
> add the application to the delegation token renewer.
> java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation,
> Service: NameNodeIP:8020, Ident: (WEBHDFS delegation token for hrt_qa)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200,
> op=RENEWDELEGATIONTOKEN, message=null
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
> at
> org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
> at org.apache.hadoop.security.token.Token.renew(Token.java:377)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
> ... 6 more
> Caused by: java.io.IOException: The error stream is null.
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:304)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:329)
> ... 24 more
> 2014-08-19 03:12:54,735 DEBUG event.AsyncDispatcher
> (AsyncDispatcher.java:
[jira] [Commented] (HDFS-7276) Limit the number of byte arrays used by DFSOutputStream
[
https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183289#comment-14183289
]
Hadoop QA commented on HDFS-7276:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676968/h7276_20141024.patch
against trunk revision e2be333.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8523//console
This message is automatically generated.
> Limit the number of byte arrays used by DFSOutputStream
> ---
>
> Key: HDFS-7276
> URL: https://issues.apache.org/jira/browse/HDFS-7276
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Attachments: h7276_20141021.patch, h7276_20141022.patch,
> h7276_20141023.patch, h7276_20141024.patch
>
>
> When there are a lot of DFSOutputStream's writing concurrently, the number of
> outstanding packets could be large. The byte arrays created by those packets
> could occupy a lot of memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7288) HDFS compiling failure on trunk
[
https://issues.apache.org/jira/browse/HDFS-7288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183284#comment-14183284
]
Chris Nauroth commented on HDFS-7288:
-
It looks like the commit of HDFS-6904 missed adding a few new classes. I've
asked [~jnp] to take a look. Thanks for reporting it, Zhijie.
> HDFS compiling failure on trunk
> ---
>
> Key: HDFS-7288
> URL: https://issues.apache.org/jira/browse/HDFS-7288
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhijie Shen
>
> See
> https://builds.apache.org/job/PreCommit-YARN-Build/5545/artifact/patchprocess/trunkJavacWarnings.txt
> {code}
> [INFO] 40 errors
> [INFO] -
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Hadoop Main SUCCESS [ 0.285 s]
> [INFO] Apache Hadoop Project POM . SUCCESS [ 0.462 s]
> [INFO] Apache Hadoop Annotations . SUCCESS [ 1.368 s]
> [INFO] Apache Hadoop Project Dist POM SUCCESS [ 0.116 s]
> [INFO] Apache Hadoop Assemblies .. SUCCESS [ 0.108 s]
> [INFO] Apache Hadoop Maven Plugins ... SUCCESS [ 2.382 s]
> [INFO] Apache Hadoop MiniKDC . SUCCESS [ 2.612 s]
> [INFO] Apache Hadoop Auth SUCCESS [ 3.365 s]
> [INFO] Apache Hadoop Auth Examples ... SUCCESS [ 1.018 s]
> [INFO] Apache Hadoop Common .. SUCCESS [ 20.312 s]
> [INFO] Apache Hadoop NFS . SUCCESS [ 3.382 s]
> [INFO] Apache Hadoop KMS . SUCCESS [ 4.705 s]
> [INFO] Apache Hadoop Common Project .. SUCCESS [ 0.029 s]
> [INFO] Apache Hadoop HDFS FAILURE [ 25.008 s]
> [INFO] Apache Hadoop HttpFS .. SKIPPED
> [INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
> [INFO] Apache Hadoop HDFS-NFS SKIPPED
> [INFO] Apache Hadoop HDFS Project SKIPPED
> [INFO] hadoop-yarn ... SKIPPED
> [INFO] hadoop-yarn-api ... SKIPPED
> [INFO] hadoop-yarn-common SKIPPED
> [INFO] hadoop-yarn-server SKIPPED
> [INFO] hadoop-yarn-server-common . SKIPPED
> [INFO] hadoop-yarn-server-nodemanager SKIPPED
> [INFO] hadoop-yarn-server-web-proxy .. SKIPPED
> [INFO] hadoop-yarn-server-applicationhistoryservice .. SKIPPED
> [INFO] hadoop-yarn-server-resourcemanager SKIPPED
> [INFO] hadoop-yarn-server-tests .. SKIPPED
> [INFO] hadoop-yarn-client SKIPPED
> [INFO] hadoop-yarn-server-sharedcachemanager . SKIPPED
> [INFO] hadoop-yarn-applications .. SKIPPED
> [INFO] hadoop-yarn-applications-distributedshell . SKIPPED
> [INFO] hadoop-yarn-applications-unmanaged-am-launcher SKIPPED
> [INFO] hadoop-yarn-site .. SKIPPED
> [INFO] hadoop-yarn-registry .. SKIPPED
> [INFO] hadoop-yarn-project ... SKIPPED
> [INFO] hadoop-mapreduce-client ... SKIPPED
> [INFO] hadoop-mapreduce-client-core .. SKIPPED
> [INFO] hadoop-mapreduce-client-common SKIPPED
> [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED
> [INFO] hadoop-mapreduce-client-app ... SKIPPED
> [INFO] hadoop-mapreduce-client-hs SKIPPED
> [INFO] hadoop-mapreduce-client-jobclient . SKIPPED
> [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED
> [INFO] hadoop-mapreduce-client-nativetask SKIPPED
> [INFO] Apache Hadoop MapReduce Examples .. SKIPPED
> [INFO] hadoop-mapreduce .. SKIPPED
> [INFO] Apache Hadoop MapReduce Streaming . SKIPPED
> [INFO] Apache Hadoop Distributed Copy SKIPPED
> [INFO] Apache Hadoop Archives SKIPPED
> [INFO] Apache Hadoop Rumen ... SKIPPED
> [INFO] Apache Hadoop Gridmix . SKIPPED
> [INFO] Apache Hadoop Data Join ... SKIPPED
> [INFO] Apache Hadoop Ant Tasks ... SKIPPED
> [INFO] Apache Hadoop Extras .. SKIPPED
> [INFO] Apache Hadoop Pipes ..
[jira] [Commented] (HDFS-6904) YARN unable to renew delegation token fetched via webhdfs due to incorrect service port
[
https://issues.apache.org/jira/browse/HDFS-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183281#comment-14183281
]
Chris Nauroth commented on HDFS-6904:
-
I think the new {{TokenKindParam}} and {{TokenServiceParam}} classes were
missed in this commit. HDFS-7288 reports that compilation is failing in trunk.
[~jnp], could you please look? Thank you.
> YARN unable to renew delegation token fetched via webhdfs due to incorrect
> service port
> ---
>
> Key: HDFS-6904
> URL: https://issues.apache.org/jira/browse/HDFS-6904
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Varun Vasudev
>Assignee: Jitendra Nath Pandey
>Priority: Critical
> Attachments: HDFS-6904.1.patch, HDFS-6904.2.patch, HDFS-6904.3.patch,
> HDFS-6904.4.patch
>
>
> YARN is unable to renew delegation tokens obtained via the WebHDFS REST API.
> The scenario is as follows -
> 1. User creates a delegation token using the WebHDFS REST API
> 2. User passes this token to YARN as part of app submission(via the YARN REST
> API)
> 3. When YARN tries to renew this delegation token, it fails because the token
> service is pointing to the RPC port but the token kind is WebHDFS.
> The exception is
> {noformat}
> 2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer
> (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to
> add the application to the delegation token renewer.
> java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation,
> Service: NameNodeIP:8020, Ident: (WEBHDFS delegation token for hrt_qa)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200,
> op=RENEWDELEGATIONTOKEN, message=null
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
> at
> org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
> at org.apache.hadoop.security.token.Token.renew(Token.java:377)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
> ... 6 more
> Caused by: java.io.IOException: The error stream is null.
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:304)
> at
> or
[jira] [Commented] (HDFS-6904) YARN unable to renew delegation token fetched via webhdfs due to incorrect service port
[
https://issues.apache.org/jira/browse/HDFS-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183283#comment-14183283
]
Jitendra Nath Pandey commented on HDFS-6904:
Committed to trunk, branch-2 and branch-2.6.
> YARN unable to renew delegation token fetched via webhdfs due to incorrect
> service port
> ---
>
> Key: HDFS-6904
> URL: https://issues.apache.org/jira/browse/HDFS-6904
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Varun Vasudev
>Assignee: Jitendra Nath Pandey
>Priority: Critical
> Attachments: HDFS-6904.1.patch, HDFS-6904.2.patch, HDFS-6904.3.patch,
> HDFS-6904.4.patch
>
>
> YARN is unable to renew delegation tokens obtained via the WebHDFS REST API.
> The scenario is as follows -
> 1. User creates a delegation token using the WebHDFS REST API
> 2. User passes this token to YARN as part of app submission(via the YARN REST
> API)
> 3. When YARN tries to renew this delegation token, it fails because the token
> service is pointing to the RPC port but the token kind is WebHDFS.
> The exception is
> {noformat}
> 2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer
> (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to
> add the application to the delegation token renewer.
> java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation,
> Service: NameNodeIP:8020, Ident: (WEBHDFS delegation token for hrt_qa)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200,
> op=RENEWDELEGATIONTOKEN, message=null
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
> at
> org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
> at org.apache.hadoop.security.token.Token.renew(Token.java:377)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
> ... 6 more
> Caused by: java.io.IOException: The error stream is null.
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:304)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:329)
> ... 24 more
> 2014-08-19 03:12:54,735 DEBUG
[jira] [Created] (HDFS-7288) HDFS compiling failure on trunk
Zhijie Shen created HDFS-7288:
-
Summary: HDFS compiling failure on trunk
Key: HDFS-7288
URL: https://issues.apache.org/jira/browse/HDFS-7288
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Zhijie Shen
See
https://builds.apache.org/job/PreCommit-YARN-Build/5545/artifact/patchprocess/trunkJavacWarnings.txt
{code}
[INFO] 40 errors
[INFO] -
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main SUCCESS [ 0.285 s]
[INFO] Apache Hadoop Project POM . SUCCESS [ 0.462 s]
[INFO] Apache Hadoop Annotations . SUCCESS [ 1.368 s]
[INFO] Apache Hadoop Project Dist POM SUCCESS [ 0.116 s]
[INFO] Apache Hadoop Assemblies .. SUCCESS [ 0.108 s]
[INFO] Apache Hadoop Maven Plugins ... SUCCESS [ 2.382 s]
[INFO] Apache Hadoop MiniKDC . SUCCESS [ 2.612 s]
[INFO] Apache Hadoop Auth SUCCESS [ 3.365 s]
[INFO] Apache Hadoop Auth Examples ... SUCCESS [ 1.018 s]
[INFO] Apache Hadoop Common .. SUCCESS [ 20.312 s]
[INFO] Apache Hadoop NFS . SUCCESS [ 3.382 s]
[INFO] Apache Hadoop KMS . SUCCESS [ 4.705 s]
[INFO] Apache Hadoop Common Project .. SUCCESS [ 0.029 s]
[INFO] Apache Hadoop HDFS FAILURE [ 25.008 s]
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS SKIPPED
[INFO] Apache Hadoop HDFS Project SKIPPED
[INFO] hadoop-yarn ... SKIPPED
[INFO] hadoop-yarn-api ... SKIPPED
[INFO] hadoop-yarn-common SKIPPED
[INFO] hadoop-yarn-server SKIPPED
[INFO] hadoop-yarn-server-common . SKIPPED
[INFO] hadoop-yarn-server-nodemanager SKIPPED
[INFO] hadoop-yarn-server-web-proxy .. SKIPPED
[INFO] hadoop-yarn-server-applicationhistoryservice .. SKIPPED
[INFO] hadoop-yarn-server-resourcemanager SKIPPED
[INFO] hadoop-yarn-server-tests .. SKIPPED
[INFO] hadoop-yarn-client SKIPPED
[INFO] hadoop-yarn-server-sharedcachemanager . SKIPPED
[INFO] hadoop-yarn-applications .. SKIPPED
[INFO] hadoop-yarn-applications-distributedshell . SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher SKIPPED
[INFO] hadoop-yarn-site .. SKIPPED
[INFO] hadoop-yarn-registry .. SKIPPED
[INFO] hadoop-yarn-project ... SKIPPED
[INFO] hadoop-mapreduce-client ... SKIPPED
[INFO] hadoop-mapreduce-client-core .. SKIPPED
[INFO] hadoop-mapreduce-client-common SKIPPED
[INFO] hadoop-mapreduce-client-shuffle ... SKIPPED
[INFO] hadoop-mapreduce-client-app ... SKIPPED
[INFO] hadoop-mapreduce-client-hs SKIPPED
[INFO] hadoop-mapreduce-client-jobclient . SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins SKIPPED
[INFO] hadoop-mapreduce-client-nativetask SKIPPED
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] hadoop-mapreduce .. SKIPPED
[INFO] Apache Hadoop MapReduce Streaming . SKIPPED
[INFO] Apache Hadoop Distributed Copy SKIPPED
[INFO] Apache Hadoop Archives SKIPPED
[INFO] Apache Hadoop Rumen ... SKIPPED
[INFO] Apache Hadoop Gridmix . SKIPPED
[INFO] Apache Hadoop Data Join ... SKIPPED
[INFO] Apache Hadoop Ant Tasks ... SKIPPED
[INFO] Apache Hadoop Extras .. SKIPPED
[INFO] Apache Hadoop Pipes ... SKIPPED
[INFO] Apache Hadoop OpenStack support ... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support . SKIPPED
[INFO] Apache Hadoop Azure support ... SKIPPED
[INFO] Apache Hadoop Client .. SKIPPED
[INFO] Apache Hadoop Mini-Cluster SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator SKIPPED
[INFO] Apache Hadoop Tools Dist ..
[jira] [Updated] (HDFS-7276) Limit the number of byte arrays used by DFSOutputStream
[ https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-7276: -- Attachment: h7276_20141024.patch Oops, I used a Java 8 API in my previous patch so that it could not be compiled in Jerkins. Here is a new patch. h7276_20141024.patch > Limit the number of byte arrays used by DFSOutputStream > --- > > Key: HDFS-7276 > URL: https://issues.apache.org/jira/browse/HDFS-7276 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h7276_20141021.patch, h7276_20141022.patch, > h7276_20141023.patch, h7276_20141024.patch > > > When there are a lot of DFSOutputStream's writing concurrently, the number of > outstanding packets could be large. The byte arrays created by those packets > could occupy a lot of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7287) The OfflineImageViewer (OIV) can output invalid XML depending on the filename
Ravi Prakash created HDFS-7287: -- Summary: The OfflineImageViewer (OIV) can output invalid XML depending on the filename Key: HDFS-7287 URL: https://issues.apache.org/jira/browse/HDFS-7287 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Ravi Prakash If the filename contains a character which is invalid in XML, TextWriterImageVisitor.write() or PBImageXmlWriter.o() prints out the string unescaped. For us this was the character 0x0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6904) YARN unable to renew delegation token fetched via webhdfs due to incorrect service port
[
https://issues.apache.org/jira/browse/HDFS-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183251#comment-14183251
]
Hudson commented on HDFS-6904:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6335 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6335/])
HDFS-6904. YARN unable to renew delegation token fetched via webhdfs due to
incorrect service port. (jitendra: rev e2be3337448ec0f6772a2ba463da376e7089b1fa)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/site/apt/WebHDFS.apt.vm
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
> YARN unable to renew delegation token fetched via webhdfs due to incorrect
> service port
> ---
>
> Key: HDFS-6904
> URL: https://issues.apache.org/jira/browse/HDFS-6904
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Varun Vasudev
>Assignee: Jitendra Nath Pandey
>Priority: Critical
> Attachments: HDFS-6904.1.patch, HDFS-6904.2.patch, HDFS-6904.3.patch,
> HDFS-6904.4.patch
>
>
> YARN is unable to renew delegation tokens obtained via the WebHDFS REST API.
> The scenario is as follows -
> 1. User creates a delegation token using the WebHDFS REST API
> 2. User passes this token to YARN as part of app submission(via the YARN REST
> API)
> 3. When YARN tries to renew this delegation token, it fails because the token
> service is pointing to the RPC port but the token kind is WebHDFS.
> The exception is
> {noformat}
> 2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer
> (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to
> add the application to the delegation token renewer.
> java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation,
> Service: NameNodeIP:8020, Ident: (WEBHDFS delegation token for hrt_qa)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200,
> op=RENEWDELEGATIONTOKEN, message=null
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
> at
> org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
> at org.apache.hadoop.security.token.Token.renew(Token.java:377)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRe
[jira] [Created] (HDFS-7286) Remove dead code in ServletUtil
Haohui Mai created HDFS-7286: Summary: Remove dead code in ServletUtil Key: HDFS-7286 URL: https://issues.apache.org/jira/browse/HDFS-7286 Project: Hadoop HDFS Issue Type: Improvement Reporter: Haohui Mai Assignee: Li Lu Priority: Minor Some code in ServletUtil is dead after the JSP UI is phased out. This jira propose to clean them up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5175) Provide clients a way to set IP header bits on connections
[
https://issues.apache.org/jira/browse/HDFS-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183233#comment-14183233
]
Chris Nauroth commented on HDFS-5175:
-
Yes, you're right. I mistakenly looked at the old httpclient dependency that
we're going to remove.
Then, if I understand correctly, we have this dependency now...
{code}
org.apache.httpcomponents
httpclient
compile
{code}
...and in addition, you're proposing that we add this dependency:
{code}
org.apache.httpcomponents
httpcomponents-client
compile
{code}
Do I have it correct now?
It's possible I'm exaggerating the impact of adding this dependency, so it
would be great to get more opinions. An email to hdfs-dev would be a good way
to call attention to it.
> Provide clients a way to set IP header bits on connections
> --
>
> Key: HDFS-5175
> URL: https://issues.apache.org/jira/browse/HDFS-5175
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.5-alpha
>Reporter: Lohit Vijayarenu
>
> It would be very helpful if we had ability for clients to set IP headers when
> they make socket connections for data transfers. We were looking into setting
> up QoS using DSCP bit and saw that there is no easy way to let clients pass
> down a specific value when clients make connection to DataNode.
> As a quick fix we did something similar to io.file.buffer.size where client
> could pass down DSCP integer value and when DFSClient opens a stream, it
> could set the value on socket using setTrafficClass
> Opening this JIRA to get more inputs from others who have had experience and
> might have already thought about this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6904) YARN unable to renew delegation token fetched via webhdfs due to incorrect service port
[
https://issues.apache.org/jira/browse/HDFS-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183177#comment-14183177
]
Varun Vasudev commented on HDFS-6904:
-
Sorry for the late comment but I did test the patch and it works as expected.
Thanks Jitendra!
> YARN unable to renew delegation token fetched via webhdfs due to incorrect
> service port
> ---
>
> Key: HDFS-6904
> URL: https://issues.apache.org/jira/browse/HDFS-6904
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Varun Vasudev
>Assignee: Jitendra Nath Pandey
>Priority: Critical
> Attachments: HDFS-6904.1.patch, HDFS-6904.2.patch, HDFS-6904.3.patch,
> HDFS-6904.4.patch
>
>
> YARN is unable to renew delegation tokens obtained via the WebHDFS REST API.
> The scenario is as follows -
> 1. User creates a delegation token using the WebHDFS REST API
> 2. User passes this token to YARN as part of app submission(via the YARN REST
> API)
> 3. When YARN tries to renew this delegation token, it fails because the token
> service is pointing to the RPC port but the token kind is WebHDFS.
> The exception is
> {noformat}
> 2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer
> (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to
> add the application to the delegation token renewer.
> java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation,
> Service: NameNodeIP:8020, Ident: (WEBHDFS delegation token for hrt_qa)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200,
> op=RENEWDELEGATIONTOKEN, message=null
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
> at
> org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
> at org.apache.hadoop.security.token.Token.renew(Token.java:377)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
> ... 6 more
> Caused by: java.io.IOException: The error stream is null.
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:304)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:329)
> ... 24
[jira] [Updated] (HDFS-7283) Bump DataNode OOM log from WARN to ERROR
[
https://issues.apache.org/jira/browse/HDFS-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7283:
-
Resolution: Fixed
Fix Version/s: 2.7.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed the patch to trunk and branch-2. Thanks [~schu] for the
contribution.
> Bump DataNode OOM log from WARN to ERROR
>
>
> Key: HDFS-7283
> URL: https://issues.apache.org/jira/browse/HDFS-7283
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.0.0-alpha
>Reporter: Stephen Chu
>Assignee: Stephen Chu
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.0
>
> Attachments: HDFS-7283.1.patch
>
>
> When the DataNode OOMs, it logs the following WARN message which should be
> bumped up to ERROR because DataNode OOM often leads to DN process abortion.
> {code}
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode is out of
> memory. Will retry in 30 seconds.
> 4751 java.lang.OutOfMemoryError: unable to create new native thread"
> {code}
> Thanks to Roland Teague for identifying this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5175) Provide clients a way to set IP header bits on connections
[
https://issues.apache.org/jira/browse/HDFS-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183148#comment-14183148
]
Ming Ma commented on HDFS-5175:
---
Thanks, Chris. Really appreciate your input.
1. Each http connection might have different DSCP values. It is a good idea to
use thread local variable so that webHDFS use it to set the DSCP value and
custom {{SocketFactory}} can access it . But yeah, the code might not be clean.
2. For httpclient dependency, per
https://issues.apache.org/jira/browse/HADOOP-10105, I assume it is on its way
out. If we keep httpclient, then we have to provide a way for webHDFS to use
httpclient given its dependency on URLConnection. For example, we can make
webHDFS configurable to use either URLConnection or httpclient. If people are
ok that, then we are good. The other option is to have webHDFS switch from
URLConnection to httpclient.
> Provide clients a way to set IP header bits on connections
> --
>
> Key: HDFS-5175
> URL: https://issues.apache.org/jira/browse/HDFS-5175
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.5-alpha
>Reporter: Lohit Vijayarenu
>
> It would be very helpful if we had ability for clients to set IP headers when
> they make socket connections for data transfers. We were looking into setting
> up QoS using DSCP bit and saw that there is no easy way to let clients pass
> down a specific value when clients make connection to DataNode.
> As a quick fix we did something similar to io.file.buffer.size where client
> could pass down DSCP integer value and when DFSClient opens a stream, it
> could set the value on socket using setTrafficClass
> Opening this JIRA to get more inputs from others who have had experience and
> might have already thought about this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7283) Bump DataNode OOM log from WARN to ERROR
[
https://issues.apache.org/jira/browse/HDFS-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183128#comment-14183128
]
Hudson commented on HDFS-7283:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6333 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6333/])
HDFS-7283. Bump DataNode OOM log from WARN to ERROR. Contributed by Stephen
Chu. (wheat9: rev b3d8a642a938da9de680b479585a7c2014b8965c)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Bump DataNode OOM log from WARN to ERROR
>
>
> Key: HDFS-7283
> URL: https://issues.apache.org/jira/browse/HDFS-7283
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.0.0-alpha
>Reporter: Stephen Chu
>Assignee: Stephen Chu
>Priority: Trivial
> Labels: supportability
> Fix For: 2.7.0
>
> Attachments: HDFS-7283.1.patch
>
>
> When the DataNode OOMs, it logs the following WARN message which should be
> bumped up to ERROR because DataNode OOM often leads to DN process abortion.
> {code}
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode is out of
> memory. Will retry in 30 seconds.
> 4751 java.lang.OutOfMemoryError: unable to create new native thread"
> {code}
> Thanks to Roland Teague for identifying this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
