[jira] [Work stopped] (HDFS-9913) DispCp doesn't use Trash with -delete option
[ https://issues.apache.org/jira/browse/HDFS-9913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-9913 stopped by John Zhuge. > DispCp doesn't use Trash with -delete option > > > Key: HDFS-9913 > URL: https://issues.apache.org/jira/browse/HDFS-9913 > Project: Hadoop HDFS > Issue Type: Bug > Components: distcp >Affects Versions: 2.6.0 >Reporter: Konstantin Shaposhnikov >Assignee: John Zhuge > > Documentation for DistCp -delete option says > ([http://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html]): > | The deletion is done by FS Shell. So the trash will be used, if it is > enable. > However it seems to be no longer the case. The latest source code > (https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java) > uses `FileSystem.delete` and trash options seems to be not applied. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10220) A large number of expired leases can make namenode unresponsive and cause failover
[ https://issues.apache.org/jira/browse/HDFS-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290640#comment-15290640 ] Nicolas Fraison commented on HDFS-10220: [~yzhangal] this is how I have understand the [~daryn] comment, having the release running every 2s during 5ms and if the release is not finished after 5ms reducing the sleep to 500ms before next release. Once the release finished well in 5ms get back to the 2s sleep. I agree that this doesn't happens on normal behaviour but on our quite 2 big clusters (40Po) with around 100k jobs running every day on each with lots of distcp syncing data between both of them I can see some times (2 to 4 times a month) some quite big release of path (around 200 to 400k) that could affect the hdfs layer (we have applied this patch so it is not the case). Even if the issues are always due to something bad from the jobs running I think that the namenode should be protective here to avoid affecting the whole cluster. In the first patch the config was a configurable parameters but there was a consensus to move it to a constant parameter. So I would like to have feedback from other guys on this to not do and redo this. Thanks for your feedback > A large number of expired leases can make namenode unresponsive and cause > failover > -- > > Key: HDFS-10220 > URL: https://issues.apache.org/jira/browse/HDFS-10220 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Nicolas Fraison >Assignee: Nicolas Fraison >Priority: Minor > Attachments: HADOOP-10220.001.patch, HADOOP-10220.002.patch, > HADOOP-10220.003.patch, HADOOP-10220.004.patch, HADOOP-10220.005.patch, > HADOOP-10220.006.patch, threaddump_zkfc.txt > > > I have faced a namenode failover due to unresponsive namenode detected by the > zkfc with lot's of WARN messages (5 millions) like this one: > _org.apache.hadoop.hdfs.StateChange: BLOCK* internalReleaseLease: All > existing blocks are COMPLETE, lease removed, file closed._ > On the threaddump taken by the zkfc there are lots of thread blocked due to a > lock. > Looking at the code, there are a lock taken by the LeaseManager.Monitor when > some lease must be released. Due to the really big number of lease to be > released the namenode has taken too many times to release them blocking all > other tasks and making the zkfc thinking that the namenode was not > available/stuck. > The idea of this patch is to limit the number of leased released each time we > check for lease so the lock won't be taken for a too long time period. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10390) Implement asynchronous setAcl/getAclStatus for DistributedFileSystem
[
https://issues.apache.org/jira/browse/HDFS-10390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290679#comment-15290679
]
Hadoop QA commented on HDFS-10390:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 8s
{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 4s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 51s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804869/HDFS-10390-HDFS-9924.005.patch
|
| JIRA Issue | HDFS-10390 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c1b0d9a8a72f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1597630 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15493/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HDFS-Build/15493/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
htt
[jira] [Commented] (HDFS-9782) RollingFileSystemSink should have configurable roll interval
[
https://issues.apache.org/jira/browse/HDFS-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290694#comment-15290694
]
Hadoop QA commented on HDFS-9782:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 44s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 4s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 12s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 20s
{color} | {color:red} root: patch generated 4 new + 14 unchanged - 5 fixed = 18
total (was 19) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 6s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 35s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 129m 3s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.hdfs.server.namenode.TestEditLog |
| | hadoop.hdfs.TestAsyncDFSRename |
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804826/HDFS-9782.008.patch |
| JIRA Issue | HDFS-9782 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5394c4f787cb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1597630 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/15490/artifact/patchprocess/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Bu
[jira] [Commented] (HDFS-9311) Support optional offload of NameNode HA service health checks to a separate RPC server.
[
https://issues.apache.org/jira/browse/HDFS-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290860#comment-15290860
]
KWON BYUNGCHANG commented on HDFS-9311:
---
I have configured below
{code}
dfs.namenode.servicerpc-address.xdev.nn1=my.host.com:8040
dfs.namenode.lifeline.rpc-address.xdev.nn1=my.host.com:8041
{code}
servicerpc port is 8040, lifeline port is 8041.
however zkfc daemon is logging using servicerpc port.
It may cause confusion.
thank you.
{code}
2016-05-19 19:18:40,566 WARN ha.HealthMonitor
(HealthMonitor.java:doHealthChecks(207)) - Service health check failed for
NameNode at my.host.com/10.114.87.91:8040: The NameNode has no resources
available
{code}
> Support optional offload of NameNode HA service health checks to a separate
> RPC server.
> ---
>
> Key: HDFS-9311
> URL: https://issues.apache.org/jira/browse/HDFS-9311
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, namenode
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: 2.8.0
>
> Attachments: HDFS-9311.001.patch, HDFS-9311.002.patch,
> HDFS-9311.003.patch
>
>
> When a NameNode is overwhelmed with load, it can lead to resource exhaustion
> of the RPC handler pools (both client-facing and service-facing).
> Eventually, this blocks the health check RPC issued from ZKFC, which triggers
> a failover. Depending on fencing configuration, the former active NameNode
> may be killed. In an overloaded situation, the new active NameNode is likely
> to suffer the same fate, because client load patterns don't change after the
> failover. This can degenerate into flapping between the 2 NameNodes without
> real recovery. If a NameNode had been killed by fencing, then it would have
> to transition through safe mode, further delaying time to recovery.
> This issue proposes a separate, optional RPC server at the NameNode for
> isolating the HA health checks. These health checks are lightweight
> operations that do not suffer from contention issues on the namesystem lock
> or other shared resources. Isolating the RPC handlers is sufficient to avoid
> this situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10434) Fix intermittent test failure of TestDataNodeErasureCodingMetrics
Rakesh R created HDFS-10434:
---
Summary: Fix intermittent test failure of
TestDataNodeErasureCodingMetrics
Key: HDFS-10434
URL: https://issues.apache.org/jira/browse/HDFS-10434
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
This jira is to fix the test case failure.
Reference :
[Build15485_TestDataNodeErasureCodingMetrics_testEcTasks|https://builds.apache.org/job/PreCommit-HDFS-Build/15485/testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeErasureCodingMetrics/testEcTasks/]
{code}
Error Message
Bad value for metric EcReconstructionTasks expected:<1> but was:<0>
Stacktrace
java.lang.AssertionError: Bad value for metric EcReconstructionTasks
expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.test.MetricsAsserts.assertCounter(MetricsAsserts.java:228)
at
org.apache.hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics.testEcTasks(TestDataNodeErasureCodingMetrics.java:92)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10434) Fix intermittent test failure of TestDataNodeErasureCodingMetrics
[
https://issues.apache.org/jira/browse/HDFS-10434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-10434:
Attachment: HDFS-10434-00.patch
> Fix intermittent test failure of TestDataNodeErasureCodingMetrics
> -
>
> Key: HDFS-10434
> URL: https://issues.apache.org/jira/browse/HDFS-10434
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-10434-00.patch
>
>
> This jira is to fix the test case failure.
> Reference :
> [Build15485_TestDataNodeErasureCodingMetrics_testEcTasks|https://builds.apache.org/job/PreCommit-HDFS-Build/15485/testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeErasureCodingMetrics/testEcTasks/]
> {code}
> Error Message
> Bad value for metric EcReconstructionTasks expected:<1> but was:<0>
> Stacktrace
> java.lang.AssertionError: Bad value for metric EcReconstructionTasks
> expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.test.MetricsAsserts.assertCounter(MetricsAsserts.java:228)
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics.testEcTasks(TestDataNodeErasureCodingMetrics.java:92)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10434) Fix intermittent test failure of TestDataNodeErasureCodingMetrics
[
https://issues.apache.org/jira/browse/HDFS-10434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-10434:
Status: Patch Available (was: Open)
> Fix intermittent test failure of TestDataNodeErasureCodingMetrics
> -
>
> Key: HDFS-10434
> URL: https://issues.apache.org/jira/browse/HDFS-10434
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-10434-00.patch
>
>
> This jira is to fix the test case failure.
> Reference :
> [Build15485_TestDataNodeErasureCodingMetrics_testEcTasks|https://builds.apache.org/job/PreCommit-HDFS-Build/15485/testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeErasureCodingMetrics/testEcTasks/]
> {code}
> Error Message
> Bad value for metric EcReconstructionTasks expected:<1> but was:<0>
> Stacktrace
> java.lang.AssertionError: Bad value for metric EcReconstructionTasks
> expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.test.MetricsAsserts.assertCounter(MetricsAsserts.java:228)
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics.testEcTasks(TestDataNodeErasureCodingMetrics.java:92)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10434) Fix intermittent test failure of TestDataNodeErasureCodingMetrics
[
https://issues.apache.org/jira/browse/HDFS-10434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290981#comment-15290981
]
Rakesh R commented on HDFS-10434:
-
[~drankye], [~libo-intel] could you please take a look at the proposed patch.
Thanks!
> Fix intermittent test failure of TestDataNodeErasureCodingMetrics
> -
>
> Key: HDFS-10434
> URL: https://issues.apache.org/jira/browse/HDFS-10434
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-10434-00.patch
>
>
> This jira is to fix the test case failure.
> Reference :
> [Build15485_TestDataNodeErasureCodingMetrics_testEcTasks|https://builds.apache.org/job/PreCommit-HDFS-Build/15485/testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeErasureCodingMetrics/testEcTasks/]
> {code}
> Error Message
> Bad value for metric EcReconstructionTasks expected:<1> but was:<0>
> Stacktrace
> java.lang.AssertionError: Bad value for metric EcReconstructionTasks
> expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.test.MetricsAsserts.assertCounter(MetricsAsserts.java:228)
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics.testEcTasks(TestDataNodeErasureCodingMetrics.java:92)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10246) Standby NameNode dfshealth.jsp Response very slow
[ https://issues.apache.org/jira/browse/HDFS-10246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291131#comment-15291131 ] Kihwal Lee commented on HDFS-10246: --- HDFS-7097 and HDFS-6763 also mitigate spikes on standby name nodes. > Standby NameNode dfshealth.jsp Response very slow > - > > Key: HDFS-10246 > URL: https://issues.apache.org/jira/browse/HDFS-10246 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 > Environment: CentOS6.3 Hadoop-2.2.0 >Reporter: zhangyubiao > Labels: bug > Attachments: stacks.txt, 屏幕快照 2016-04-07 下午6.04.36.png > > > HDFS Standby NameNode dfshealth.jsp Response very slow -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10417) Improve error message from checkBlockLocalPathAccess
[
https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291199#comment-15291199
]
Hadoop QA commented on HDFS-10417:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
5s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 25s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new +
193 unchanged - 0 fixed = 195 total (was 193) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 30s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 1s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804519/HDFS-10417.001.patch |
| JIRA Issue | HDFS-10417 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2d3b60306ac8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d4274c6 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/15494/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15494/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HDFS-Build/15494/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15494/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15494/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Improve error message from checkBlockL
[jira] [Commented] (HDFS-9924) [umbrella] Asynchronous HDFS Access
[ https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291526#comment-15291526 ] Xiaobing Zhou commented on HDFS-9924: - I need to clarify 'No' meant sync/async calls actually go to different RPC path. For your question, of course yes. [~mingma] > [umbrella] Asynchronous HDFS Access > --- > > Key: HDFS-9924 > URL: https://issues.apache.org/jira/browse/HDFS-9924 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Reporter: Tsz Wo Nicholas Sze >Assignee: Xiaobing Zhou > Attachments: AsyncHdfs20160510.pdf > > > This is an umbrella JIRA for supporting Asynchronous HDFS Access. > Currently, all the API methods are blocking calls -- the caller is blocked > until the method returns. It is very slow if a client makes a large number > of independent calls in a single thread since each call has to wait until the > previous call is finished. It is inefficient if a client needs to create a > large number of threads to invoke the calls. > We propose adding a new API to support asynchronous calls, i.e. the caller is > not blocked. The methods in the new API immediately return a Java Future > object. The return value can be obtained by the usual Future.get() method. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10417) Improve error message from checkBlockLocalPathAccess
[
https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyin Xu updated HDFS-10417:
--
Attachment: HDFS-10417.002.patch
the new patch that follow the style rule
> Improve error message from checkBlockLocalPathAccess
>
>
> Key: HDFS-10417
> URL: https://issues.apache.org/jira/browse/HDFS-10417
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-10417.000.patch, HDFS-10417.001.patch,
> HDFS-10417.002.patch
>
>
> The exception msg thrown by {{checkBlockLocalPathAccess}} is very specific to
> the implementation detail. It's really hard for users to understand it unless
> she reads and understands the code.
> The code is shown as follows:
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private void checkBlockLocalPathAccess() throws IOException {
> checkKerberosAuthMethod("getBlockLocalPathInfo()");
> String currentUser =
> UserGroupInformation.getCurrentUser().getShortUserName();
> if (!usersWithLocalPathAccess.contains(currentUser)) {
> throw new AccessControlException(
> "Can't continue with getBlockLocalPathInfo() "
> + "authorization. The user " + currentUser
> + " is not allowed to call getBlockLocalPathInfo");
> }
> }
> {code}
> (basically she needs to understand the code logic of getBlockLocalPathInfo)
> \\
> Note that {{usersWithLocalPathAccess}} is a *private final* purely coming
> from the configuration settings of {{dfs.block.local-path-access.user}},
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private final List usersWithLocalPathAccess;
>
> this.usersWithLocalPathAccess = Arrays.asList(
>
> conf.getTrimmedStrings(DFSConfigKeys.DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY));
> {code}
> In other word, the checking fails simply because the current user is not
> specified in the configuration setting of
> {{dfs.block.local-path-access.user}}. The log message should be much more
> clearer to make it easy for users to take actions, as demonstrated in the
> attached patch.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10385) LocalFileSystem rename() function should return false when destination file exists
[ https://issues.apache.org/jira/browse/HDFS-10385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291630#comment-15291630 ] Andras Bokor commented on HDFS-10385: - Previously, I had a discussion with [~cnauroth] and [~ste...@apache.org] about changing rename behaviors and finally that JIRA was resolved with "Later" resolution. Please check the HADOOP-13082 for details. Even if we change behaviors it should be part of a bigger and more complex change because there are lot of differences between rename behaviors in different FS implementations. Based on that this should be closed either. What do you guys think? > LocalFileSystem rename() function should return false when destination file > exists > -- > > Key: HDFS-10385 > URL: https://issues.apache.org/jira/browse/HDFS-10385 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Affects Versions: 2.6.0 >Reporter: Aihua Xu >Assignee: Xiaobing Zhou > > Currently rename() of LocalFileSystem returns true and renames successfully > when the destination file exists. That seems to have different behavior from > DFSFileSystem. > If they can have the same behavior, then we can use one call to do rename > rather than checking if destination exists and then making rename() call. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10363) Ozone: Introduce new config keys for SCM service endpoints
[ https://issues.apache.org/jira/browse/HDFS-10363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291653#comment-15291653 ] Anu Engineer commented on HDFS-10363: - +1, LGTM. [~arpitagarwal] Thanks for the update. > Ozone: Introduce new config keys for SCM service endpoints > -- > > Key: HDFS-10363 > URL: https://issues.apache.org/jira/browse/HDFS-10363 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10363-HDFS-7240.02.patch, > HDFS-10363-HDFS-7240.03.patch, HDFS-10363-HDFS-7240.04.patch, > HDFS-10363-HDFS-7240.05.patch, HDFS-10363.01.patch, HDFS-10363.02.patch, > OzoneScmEndpointconfiguration.pdf, ozone-site.xml > > > The SCM should have its own config keys to specify service addresses. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10363) Ozone: Introduce new config keys for SCM service endpoints
[ https://issues.apache.org/jira/browse/HDFS-10363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-10363: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) [~arpitagarwal] Thanks for the contribution. I have committed this to the feature branch. > Ozone: Introduce new config keys for SCM service endpoints > -- > > Key: HDFS-10363 > URL: https://issues.apache.org/jira/browse/HDFS-10363 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10363-HDFS-7240.02.patch, > HDFS-10363-HDFS-7240.03.patch, HDFS-10363-HDFS-7240.04.patch, > HDFS-10363-HDFS-7240.05.patch, HDFS-10363.01.patch, HDFS-10363.02.patch, > OzoneScmEndpointconfiguration.pdf, ozone-site.xml > > > The SCM should have its own config keys to specify service addresses. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10432) "dfs.namenode.hosts" and "dfs.namenode.host.exclude" deprecated?
[ https://issues.apache.org/jira/browse/HDFS-10432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291690#comment-15291690 ] Rushabh S Shah commented on HDFS-10432: --- bq. But, the parameter "dfs.namenode.hosts" and "dfs.namenode.host.exclude" don´t appear in "hdfs-default.xml" file. Both of these configs are removed from code HDFS-9005. Instead the contributors added new config: dfs.namenode.hosts.provider.classname > "dfs.namenode.hosts" and "dfs.namenode.host.exclude" deprecated? > - > > Key: HDFS-10432 > URL: https://issues.apache.org/jira/browse/HDFS-10432 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.6.2 >Reporter: Ernesto Peralta >Priority: Trivial > > Hello, > -https://hadoop.apache.org/docs/r2.6.2/hadoop-project-dist/hadoop-common/ClusterSetup.html > > we can see In the configurations for NameNode from "hdfs-site.xml", the > parameters "dfs.namenode.hosts" and "dfs.namenode.hosts.exclude". > But, the parameter "dfs.namenode.hosts" and "dfs.namenode.host.exclude" don´t > appear in "hdfs-default.xml" file. > Are the parameters "dfs.hosts" and "dfs.hosts.exclude" more adequate and > right? Are the parameters "dfs.namenode.hosts" and > "dfs.namenode.host.exclude" deprecated? > -hdfs-default.xml, parameters: > http://hadoop.apache.org/docs/r2.6.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10434) Fix intermittent test failure of TestDataNodeErasureCodingMetrics
[
https://issues.apache.org/jira/browse/HDFS-10434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291910#comment-15291910
]
Hadoop QA commented on HDFS-10434:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 18s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 24s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.web.TestWebHDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804920/HDFS-10434-00.patch |
| JIRA Issue | HDFS-10434 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux aaeed431c00e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 141873c |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15495/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HDFS-Build/15495/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15495/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15495/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Fix intermittent test failure of TestDataNodeErasureCodingMetrics
> -
>
> Key: HDFS-10434
> URL: https://issues.apache.org/jira/browse/HDF
[jira] [Updated] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[ https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10420: - Attachment: HDFS-10420.01.patch v01 patch for Jenkins run. It fixes a bunch of previously failing tests. TestOzoneContainer and TestBlockPoolManager are still known to be broken. > Fix Ozone unit tests to use MiniOzoneCluster > > > Key: HDFS-10420 > URL: https://issues.apache.org/jira/browse/HDFS-10420 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10420.01.patch > > > [~cnauroth] added MiniOzoneCluster support to the Ozone feature branch. We > should update all the Ozone tests to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[ https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10420: - Attachment: (was: HDFS-10420.01.patch) > Fix Ozone unit tests to use MiniOzoneCluster > > > Key: HDFS-10420 > URL: https://issues.apache.org/jira/browse/HDFS-10420 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10420-HDFS-7240.01.patch > > > [~cnauroth] added MiniOzoneCluster support to the Ozone feature branch. We > should update all the Ozone tests to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[ https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10420: - Attachment: HDFS-10420-HDFS-7240.01.patch > Fix Ozone unit tests to use MiniOzoneCluster > > > Key: HDFS-10420 > URL: https://issues.apache.org/jira/browse/HDFS-10420 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10420-HDFS-7240.01.patch > > > [~cnauroth] added MiniOzoneCluster support to the Ozone feature branch. We > should update all the Ozone tests to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[ https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10420: - Status: Patch Available (was: In Progress) > Fix Ozone unit tests to use MiniOzoneCluster > > > Key: HDFS-10420 > URL: https://issues.apache.org/jira/browse/HDFS-10420 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10420-HDFS-7240.01.patch > > > [~cnauroth] added MiniOzoneCluster support to the Ozone feature branch. We > should update all the Ozone tests to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10417) Improve error message from checkBlockLocalPathAccess
[
https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyin Xu updated HDFS-10417:
--
Attachment: (was: HDFS-10417.002.patch)
> Improve error message from checkBlockLocalPathAccess
>
>
> Key: HDFS-10417
> URL: https://issues.apache.org/jira/browse/HDFS-10417
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-10417.000.patch, HDFS-10417.001.patch
>
>
> The exception msg thrown by {{checkBlockLocalPathAccess}} is very specific to
> the implementation detail. It's really hard for users to understand it unless
> she reads and understands the code.
> The code is shown as follows:
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private void checkBlockLocalPathAccess() throws IOException {
> checkKerberosAuthMethod("getBlockLocalPathInfo()");
> String currentUser =
> UserGroupInformation.getCurrentUser().getShortUserName();
> if (!usersWithLocalPathAccess.contains(currentUser)) {
> throw new AccessControlException(
> "Can't continue with getBlockLocalPathInfo() "
> + "authorization. The user " + currentUser
> + " is not allowed to call getBlockLocalPathInfo");
> }
> }
> {code}
> (basically she needs to understand the code logic of getBlockLocalPathInfo)
> \\
> Note that {{usersWithLocalPathAccess}} is a *private final* purely coming
> from the configuration settings of {{dfs.block.local-path-access.user}},
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private final List usersWithLocalPathAccess;
>
> this.usersWithLocalPathAccess = Arrays.asList(
>
> conf.getTrimmedStrings(DFSConfigKeys.DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY));
> {code}
> In other word, the checking fails simply because the current user is not
> specified in the configuration setting of
> {{dfs.block.local-path-access.user}}. The log message should be much more
> clearer to make it easy for users to take actions, as demonstrated in the
> attached patch.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10417) Improve error message from checkBlockLocalPathAccess
[
https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyin Xu updated HDFS-10417:
--
Attachment: HDFS-10417.002.patch
> Improve error message from checkBlockLocalPathAccess
>
>
> Key: HDFS-10417
> URL: https://issues.apache.org/jira/browse/HDFS-10417
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-10417.000.patch, HDFS-10417.001.patch,
> HDFS-10417.002.patch
>
>
> The exception msg thrown by {{checkBlockLocalPathAccess}} is very specific to
> the implementation detail. It's really hard for users to understand it unless
> she reads and understands the code.
> The code is shown as follows:
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private void checkBlockLocalPathAccess() throws IOException {
> checkKerberosAuthMethod("getBlockLocalPathInfo()");
> String currentUser =
> UserGroupInformation.getCurrentUser().getShortUserName();
> if (!usersWithLocalPathAccess.contains(currentUser)) {
> throw new AccessControlException(
> "Can't continue with getBlockLocalPathInfo() "
> + "authorization. The user " + currentUser
> + " is not allowed to call getBlockLocalPathInfo");
> }
> }
> {code}
> (basically she needs to understand the code logic of getBlockLocalPathInfo)
> \\
> Note that {{usersWithLocalPathAccess}} is a *private final* purely coming
> from the configuration settings of {{dfs.block.local-path-access.user}},
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private final List usersWithLocalPathAccess;
>
> this.usersWithLocalPathAccess = Arrays.asList(
>
> conf.getTrimmedStrings(DFSConfigKeys.DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY));
> {code}
> In other word, the checking fails simply because the current user is not
> specified in the configuration setting of
> {{dfs.block.local-path-access.user}}. The log message should be much more
> clearer to make it easy for users to take actions, as demonstrated in the
> attached patch.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10417) Improve error message from checkBlockLocalPathAccess
[
https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291933#comment-15291933
]
Hadoop QA commented on HDFS-10417:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 19s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 52s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestFsDatasetCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804983/HDFS-10417.002.patch |
| JIRA Issue | HDFS-10417 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2768d1cfd6d0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 141873c |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/15496/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15496/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HDFS-Build/15496/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15496/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15496/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Improve error message from checkBlockLocalPathAccess
>
[jira] [Updated] (HDFS-10417) Improve error message from checkBlockLocalPathAccess
[
https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyin Xu updated HDFS-10417:
--
Attachment: (was: HDFS-10417.002.patch)
> Improve error message from checkBlockLocalPathAccess
>
>
> Key: HDFS-10417
> URL: https://issues.apache.org/jira/browse/HDFS-10417
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-10417.000.patch, HDFS-10417.001.patch,
> HDFS-10417.002.patch
>
>
> The exception msg thrown by {{checkBlockLocalPathAccess}} is very specific to
> the implementation detail. It's really hard for users to understand it unless
> she reads and understands the code.
> The code is shown as follows:
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private void checkBlockLocalPathAccess() throws IOException {
> checkKerberosAuthMethod("getBlockLocalPathInfo()");
> String currentUser =
> UserGroupInformation.getCurrentUser().getShortUserName();
> if (!usersWithLocalPathAccess.contains(currentUser)) {
> throw new AccessControlException(
> "Can't continue with getBlockLocalPathInfo() "
> + "authorization. The user " + currentUser
> + " is not allowed to call getBlockLocalPathInfo");
> }
> }
> {code}
> (basically she needs to understand the code logic of getBlockLocalPathInfo)
> \\
> Note that {{usersWithLocalPathAccess}} is a *private final* purely coming
> from the configuration settings of {{dfs.block.local-path-access.user}},
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private final List usersWithLocalPathAccess;
>
> this.usersWithLocalPathAccess = Arrays.asList(
>
> conf.getTrimmedStrings(DFSConfigKeys.DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY));
> {code}
> In other word, the checking fails simply because the current user is not
> specified in the configuration setting of
> {{dfs.block.local-path-access.user}}. The log message should be much more
> clearer to make it easy for users to take actions, as demonstrated in the
> attached patch.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10417) Improve error message from checkBlockLocalPathAccess
[
https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyin Xu updated HDFS-10417:
--
Attachment: HDFS-10417.002.patch
> Improve error message from checkBlockLocalPathAccess
>
>
> Key: HDFS-10417
> URL: https://issues.apache.org/jira/browse/HDFS-10417
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-10417.000.patch, HDFS-10417.001.patch,
> HDFS-10417.002.patch
>
>
> The exception msg thrown by {{checkBlockLocalPathAccess}} is very specific to
> the implementation detail. It's really hard for users to understand it unless
> she reads and understands the code.
> The code is shown as follows:
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private void checkBlockLocalPathAccess() throws IOException {
> checkKerberosAuthMethod("getBlockLocalPathInfo()");
> String currentUser =
> UserGroupInformation.getCurrentUser().getShortUserName();
> if (!usersWithLocalPathAccess.contains(currentUser)) {
> throw new AccessControlException(
> "Can't continue with getBlockLocalPathInfo() "
> + "authorization. The user " + currentUser
> + " is not allowed to call getBlockLocalPathInfo");
> }
> }
> {code}
> (basically she needs to understand the code logic of getBlockLocalPathInfo)
> \\
> Note that {{usersWithLocalPathAccess}} is a *private final* purely coming
> from the configuration settings of {{dfs.block.local-path-access.user}},
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private final List usersWithLocalPathAccess;
>
> this.usersWithLocalPathAccess = Arrays.asList(
>
> conf.getTrimmedStrings(DFSConfigKeys.DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY));
> {code}
> In other word, the checking fails simply because the current user is not
> specified in the configuration setting of
> {{dfs.block.local-path-access.user}}. The log message should be much more
> clearer to make it easy for users to take actions, as demonstrated in the
> attached patch.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[ https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10420: - Attachment: HDFS-10420-HDFS-7240.02.patch Rebased patch and fixed TestOzoneContainer. TestBlockPoolManager will be fixed in a separate Jira. > Fix Ozone unit tests to use MiniOzoneCluster > > > Key: HDFS-10420 > URL: https://issues.apache.org/jira/browse/HDFS-10420 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10420-HDFS-7240.01.patch, > HDFS-10420-HDFS-7240.02.patch > > > [~cnauroth] added MiniOzoneCluster support to the Ozone feature branch. We > should update all the Ozone tests to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[
https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292018#comment-15292018
]
Hadoop QA commented on HDFS-10420:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 54s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 3m 24s
{color} | {color:red} root in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 9s
{color} | {color:red} hadoop-hdfs in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 27s
{color} | {color:red} branch/hadoop-hdfs-project/hadoop-hdfs no findbugs output
file (hadoop-hdfs-project/hadoop-hdfs/target/findbugsXml.xml) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 38s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 39s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91 with JDK v1.8.0_91
generated 1 new + 34 unchanged - 36 fixed = 35 total (was 70) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 24s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101 with JDK v1.7.0_101
generated 2 new + 35 unchanged - 39 fixed = 37 total (was 74) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 21s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new +
1 unchanged - 2 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s
{color} | {color:red} patch/hadoop-hdfs-project/hadoop-hdfs no findbugs output
file (hadoop-hdfs-project/hadoop-hdfs/target/findbugsXml.xml) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 7s
{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91 with JDK
v1.8.0_91 generated 100 new + 0 unchanged - 14 fixed = 100 total (was 14)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 24s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 37s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {colo
[jira] [Commented] (HDFS-10399) DiskBalancer: Add JMX for DiskBalancer
[
https://issues.apache.org/jira/browse/HDFS-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292020#comment-15292020
]
Arpit Agarwal commented on HDFS-10399:
--
+1 thanks Anu.
Typo in parseJson Javadoc, feel free to fix while committing it.
{code}
* Returns a DiskBalancerWorkStatus object from the Json reXpresentation.
{code}
> DiskBalancer: Add JMX for DiskBalancer
> --
>
> Key: HDFS-10399
> URL: https://issues.apache.org/jira/browse/HDFS-10399
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Attachments: HDFS-10399-HDFS-1312.001.patch
>
>
> Expose diskbalancer status via JMX
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[ https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10420: - Hadoop Flags: (was: Reviewed) > Fix Ozone unit tests to use MiniOzoneCluster > > > Key: HDFS-10420 > URL: https://issues.apache.org/jira/browse/HDFS-10420 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10420-HDFS-7240.01.patch, > HDFS-10420-HDFS-7240.02.patch > > > [~cnauroth] added MiniOzoneCluster support to the Ozone feature branch. We > should update all the Ozone tests to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[ https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10420: - Hadoop Flags: Reviewed > Fix Ozone unit tests to use MiniOzoneCluster > > > Key: HDFS-10420 > URL: https://issues.apache.org/jira/browse/HDFS-10420 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10420-HDFS-7240.01.patch, > HDFS-10420-HDFS-7240.02.patch > > > [~cnauroth] added MiniOzoneCluster support to the Ozone feature branch. We > should update all the Ozone tests to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10399) DiskBalancer: Add JMX for DiskBalancer
[ https://issues.apache.org/jira/browse/HDFS-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10399: - Hadoop Flags: Reviewed > DiskBalancer: Add JMX for DiskBalancer > -- > > Key: HDFS-10399 > URL: https://issues.apache.org/jira/browse/HDFS-10399 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-10399-HDFS-1312.001.patch > > > Expose diskbalancer status via JMX -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10435) Document property fs.getspaceused.classname
Wei-Chiu Chuang created HDFS-10435:
--
Summary: Document property fs.getspaceused.classname
Key: HDFS-10435
URL: https://issues.apache.org/jira/browse/HDFS-10435
Project: Hadoop HDFS
Issue Type: Improvement
Components: fs
Affects Versions: 2.8.0
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang
Priority: Minor
HADOOP-12973 introduced a new property {{fs.getspaceused.classname}} which
makes it configurable to change the mechanism for estimating disk usage. This
is great work, thanks [~eclark]!
In Hadoop convention, this property should be declared as a string constant (or
in Java's terminology, public static final variable), and be documented in
hdfs-default.xml
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10415) TestDistributedFileSystem#testDFSCloseOrdering() fails on branch-2
[
https://issues.apache.org/jira/browse/HDFS-10415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292024#comment-15292024
]
Mingliang Liu commented on HDFS-10415:
--
When [HADOOP-9418] was committed to {{trunk}} and {{branch-2}} the overridden
{{MyDistributedFileSystem#delete()}} was not in {{branch-2}}, which is the only
difference between the two commits. I don't have too much context about this,
and boldly assume this as an omission.
As this case is for test only and hardly shows up in real systems, I don't have
strong preference (which is listed though) regarding the possible fixes we
discussed here.
# Make {{MyDistributedFileSystem}} override {{delete()}} in {{branch-2}} as in
{{trunk}} (see patch v1)
# Spy instead of mocking (see patch v0) for both {{trunk}} and {{branch-2}}
# Construct {{storageStatistics}} as we construct {{statistics}}
> TestDistributedFileSystem#testDFSCloseOrdering() fails on branch-2
> --
>
> Key: HDFS-10415
> URL: https://issues.apache.org/jira/browse/HDFS-10415
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.9.0
> Environment: jenkins
>Reporter: Sangjin Lee
>Assignee: Mingliang Liu
> Attachments: HDFS-10415-branch-2.000.patch,
> HDFS-10415-branch-2.001.patch, HDFS-10415.000.patch
>
>
> {noformat}
> Tests run: 24, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 51.096 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestDistributedFileSystem
> testDFSCloseOrdering(org.apache.hadoop.hdfs.TestDistributedFileSystem) Time
> elapsed: 0.045 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:790)
> at
> org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1417)
> at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2084)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1187)
> at
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testDFSCloseOrdering(TestDistributedFileSystem.java:217)
> {noformat}
> This is with Java 8 on Mac. It passes fine on trunk. I haven't tried other
> combinations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10402) DiskBalancer: Add QueryStatus command
[ https://issues.apache.org/jira/browse/HDFS-10402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292027#comment-15292027 ] Arpit Agarwal commented on HDFS-10402: -- +1, thanks Anu. I assume this will need to be rebased when HDFS-10399 is committed. > DiskBalancer: Add QueryStatus command > - > > Key: HDFS-10402 > URL: https://issues.apache.org/jira/browse/HDFS-10402 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-1312 > > Attachments: HDFS-10402-HDFS-1312.001.patch > > > Adds QueryStatus which gets disk balancer status from a specific node. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10402) DiskBalancer: Add QueryStatus command
[ https://issues.apache.org/jira/browse/HDFS-10402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10402: - Hadoop Flags: Reviewed > DiskBalancer: Add QueryStatus command > - > > Key: HDFS-10402 > URL: https://issues.apache.org/jira/browse/HDFS-10402 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-1312 > > Attachments: HDFS-10402-HDFS-1312.001.patch > > > Adds QueryStatus which gets disk balancer status from a specific node. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10435) Document property fs.getspaceused.classname
[
https://issues.apache.org/jira/browse/HDFS-10435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-10435:
---
Description:
HADOOP-12973 introduced a new property {{fs.getspaceused.classname}} which
makes it configurable to change the mechanism for estimating disk usage. This
is great work, thanks [~eclark]!
In Hadoop convention, this property should be declared as a string constant (or
in Java's terminology, public static final variable), and be documented in
core-default.xml
was:
HADOOP-12973 introduced a new property {{fs.getspaceused.classname}} which
makes it configurable to change the mechanism for estimating disk usage. This
is great work, thanks [~eclark]!
In Hadoop convention, this property should be declared as a string constant (or
in Java's terminology, public static final variable), and be documented in
hdfs-default.xml
> Document property fs.getspaceused.classname
> ---
>
> Key: HDFS-10435
> URL: https://issues.apache.org/jira/browse/HDFS-10435
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
>
> HADOOP-12973 introduced a new property {{fs.getspaceused.classname}} which
> makes it configurable to change the mechanism for estimating disk usage. This
> is great work, thanks [~eclark]!
> In Hadoop convention, this property should be declared as a string constant
> (or in Java's terminology, public static final variable), and be documented
> in core-default.xml
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10436) dfs.block.access.token.enable should default on when security is !simple
Allen Wittenauer created HDFS-10436: --- Summary: dfs.block.access.token.enable should default on when security is !simple Key: HDFS-10436 URL: https://issues.apache.org/jira/browse/HDFS-10436 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 3.0.0-alpha1 Reporter: Allen Wittenauer Unless there is a valid configuration where dfs.block.access.token.enable is off and security is on, then rather than shutdown we should just enable the block access tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9547) DiskBalancer : Add user documentation
[
https://issues.apache.org/jira/browse/HDFS-9547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292057#comment-15292057
]
Arpit Agarwal commented on HDFS-9547:
-
This is excellent. A few nitpicky comments:
# The documentation should be under the hadoop-hdfs project.
# We should add a link to the DiskBalancer documentation from the
HDFSCommands.html page. Alternatively it can just be a sub-section in
HDFSCommands.html like balancer, instead of a new page.
# For commands the font is quite small. We could use the same font used by the
HDFSCommands.html page.
# Document verbose option for query
# The following statement could link to the balancer docs so the difference is
clearer to someone not familiar with the balancer. i.e. link to
HDFSCommands.html#balancer
{quote}
This tool is different from balancer which takes care of cluster-wide data
balancing.
{quote}
> DiskBalancer : Add user documentation
> -
>
> Key: HDFS-9547
> URL: https://issues.apache.org/jira/browse/HDFS-9547
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Attachments: HDFS-9547-HDFS-1312.001.patch
>
>
> Write diskbalancer.md since this is a new tool and explain the usage with
> examples.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10403) DiskBalancer: Add cancel command
[ https://issues.apache.org/jira/browse/HDFS-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292085#comment-15292085 ] Arpit Agarwal commented on HDFS-10403: -- Thanks for the patch Anu. It looks like cancel only works with a plan file. We could also allow canceling with a target node and plan ID. If the admin loses the plan file, he can lookup the plan ID for a node via query then issue cancel. > DiskBalancer: Add cancel command > - > > Key: HDFS-10403 > URL: https://issues.apache.org/jira/browse/HDFS-10403 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-1312 > > Attachments: HDFS-10403-HDFS-1312.001.patch > > > Allows user to cancel an on-going disk balancing operation -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10417) Improve error message from checkBlockLocalPathAccess
[
https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292090#comment-15292090
]
Hadoop QA commented on HDFS-10417:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 12s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 33s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestFileCreationDelete |
| | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805022/HDFS-10417.002.patch |
| JIRA Issue | HDFS-10417 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3bfa86c98c60 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 141873c |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15498/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HDFS-Build/15498/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15498/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15498/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Improve error message from checkBlockLocalPathAccess
>
>
> Key: HDFS-10417
> URL: https://issues.apache.org/jira/browse/H
[jira] [Updated] (HDFS-10390) Implement asynchronous setAcl/getAclStatus for DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10390: - Attachment: HDFS-10390-HDFS-9924.006.patch > Implement asynchronous setAcl/getAclStatus for DistributedFileSystem > > > Key: HDFS-10390 > URL: https://issues.apache.org/jira/browse/HDFS-10390 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10390-HDFS-9924.000.patch, > HDFS-10390-HDFS-9924.001.patch, HDFS-10390-HDFS-9924.002.patch, > HDFS-10390-HDFS-9924.003.patch, HDFS-10390-HDFS-9924.004.patch, > HDFS-10390-HDFS-9924.005.patch, HDFS-10390-HDFS-9924.006.patch > > > This is proposed to implement asynchronous setAcl/getAclStatus. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10402) DiskBalancer: Add QueryStatus command
[ https://issues.apache.org/jira/browse/HDFS-10402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292099#comment-15292099 ] Arpit Agarwal commented on HDFS-10402: -- Also a trivial typo that can be fixed at commit time _Constructs ExecuteCommand_ --> _Constructs QueryCommand_. > DiskBalancer: Add QueryStatus command > - > > Key: HDFS-10402 > URL: https://issues.apache.org/jira/browse/HDFS-10402 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: HDFS-1312 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-1312 > > Attachments: HDFS-10402-HDFS-1312.001.patch > > > Adds QueryStatus which gets disk balancer status from a specific node. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10390) Implement asynchronous setAcl/getAclStatus for DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292100#comment-15292100 ] Xiaobing Zhou commented on HDFS-10390: -- v006 patch is posted. 1. in TestAsyncDFS#outOfOrderPut, use Collections.Shuffle to simulate out of order, e.g. Collections.shuffle(futureList); 2. in TestAsyncDFSRename, prolong timeout instead of removing it. 3. In TestAsyncDFS, remove tests of aggressive async calls, but keep tests of conservative ones by limiting max number of async calls. Latter works well ensures former being true. This avoids doubling tests for every API. > Implement asynchronous setAcl/getAclStatus for DistributedFileSystem > > > Key: HDFS-10390 > URL: https://issues.apache.org/jira/browse/HDFS-10390 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10390-HDFS-9924.000.patch, > HDFS-10390-HDFS-9924.001.patch, HDFS-10390-HDFS-9924.002.patch, > HDFS-10390-HDFS-9924.003.patch, HDFS-10390-HDFS-9924.004.patch, > HDFS-10390-HDFS-9924.005.patch, HDFS-10390-HDFS-9924.006.patch > > > This is proposed to implement asynchronous setAcl/getAclStatus. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10417) Improve error message from checkBlockLocalPathAccess
[
https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292118#comment-15292118
]
Kihwal Lee commented on HDFS-10417:
---
+1 for the latest patch.
> Improve error message from checkBlockLocalPathAccess
>
>
> Key: HDFS-10417
> URL: https://issues.apache.org/jira/browse/HDFS-10417
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-10417.000.patch, HDFS-10417.001.patch,
> HDFS-10417.002.patch
>
>
> The exception msg thrown by {{checkBlockLocalPathAccess}} is very specific to
> the implementation detail. It's really hard for users to understand it unless
> she reads and understands the code.
> The code is shown as follows:
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private void checkBlockLocalPathAccess() throws IOException {
> checkKerberosAuthMethod("getBlockLocalPathInfo()");
> String currentUser =
> UserGroupInformation.getCurrentUser().getShortUserName();
> if (!usersWithLocalPathAccess.contains(currentUser)) {
> throw new AccessControlException(
> "Can't continue with getBlockLocalPathInfo() "
> + "authorization. The user " + currentUser
> + " is not allowed to call getBlockLocalPathInfo");
> }
> }
> {code}
> (basically she needs to understand the code logic of getBlockLocalPathInfo)
> \\
> Note that {{usersWithLocalPathAccess}} is a *private final* purely coming
> from the configuration settings of {{dfs.block.local-path-access.user}},
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private final List usersWithLocalPathAccess;
>
> this.usersWithLocalPathAccess = Arrays.asList(
>
> conf.getTrimmedStrings(DFSConfigKeys.DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY));
> {code}
> In other word, the checking fails simply because the current user is not
> specified in the configuration setting of
> {{dfs.block.local-path-access.user}}. The log message should be much more
> clearer to make it easy for users to take actions, as demonstrated in the
> attached patch.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10417) Improve error message from checkBlockLocalPathAccess
[
https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292121#comment-15292121
]
Kihwal Lee commented on HDFS-10417:
---
Committed this to trunk, branch-2 and branch-2.8. Thanks for the patch,
[~tianyin].
> Improve error message from checkBlockLocalPathAccess
>
>
> Key: HDFS-10417
> URL: https://issues.apache.org/jira/browse/HDFS-10417
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-10417.000.patch, HDFS-10417.001.patch,
> HDFS-10417.002.patch
>
>
> The exception msg thrown by {{checkBlockLocalPathAccess}} is very specific to
> the implementation detail. It's really hard for users to understand it unless
> she reads and understands the code.
> The code is shown as follows:
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private void checkBlockLocalPathAccess() throws IOException {
> checkKerberosAuthMethod("getBlockLocalPathInfo()");
> String currentUser =
> UserGroupInformation.getCurrentUser().getShortUserName();
> if (!usersWithLocalPathAccess.contains(currentUser)) {
> throw new AccessControlException(
> "Can't continue with getBlockLocalPathInfo() "
> + "authorization. The user " + currentUser
> + " is not allowed to call getBlockLocalPathInfo");
> }
> }
> {code}
> (basically she needs to understand the code logic of getBlockLocalPathInfo)
> \\
> Note that {{usersWithLocalPathAccess}} is a *private final* purely coming
> from the configuration settings of {{dfs.block.local-path-access.user}},
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private final List usersWithLocalPathAccess;
>
> this.usersWithLocalPathAccess = Arrays.asList(
>
> conf.getTrimmedStrings(DFSConfigKeys.DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY));
> {code}
> In other word, the checking fails simply because the current user is not
> specified in the configuration setting of
> {{dfs.block.local-path-access.user}}. The log message should be much more
> clearer to make it easy for users to take actions, as demonstrated in the
> attached patch.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10417) Improve error message from checkBlockLocalPathAccess
[
https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-10417:
--
Status: In Progress (was: Patch Available)
> Improve error message from checkBlockLocalPathAccess
>
>
> Key: HDFS-10417
> URL: https://issues.apache.org/jira/browse/HDFS-10417
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-10417.000.patch, HDFS-10417.001.patch,
> HDFS-10417.002.patch
>
>
> The exception msg thrown by {{checkBlockLocalPathAccess}} is very specific to
> the implementation detail. It's really hard for users to understand it unless
> she reads and understands the code.
> The code is shown as follows:
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private void checkBlockLocalPathAccess() throws IOException {
> checkKerberosAuthMethod("getBlockLocalPathInfo()");
> String currentUser =
> UserGroupInformation.getCurrentUser().getShortUserName();
> if (!usersWithLocalPathAccess.contains(currentUser)) {
> throw new AccessControlException(
> "Can't continue with getBlockLocalPathInfo() "
> + "authorization. The user " + currentUser
> + " is not allowed to call getBlockLocalPathInfo");
> }
> }
> {code}
> (basically she needs to understand the code logic of getBlockLocalPathInfo)
> \\
> Note that {{usersWithLocalPathAccess}} is a *private final* purely coming
> from the configuration settings of {{dfs.block.local-path-access.user}},
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private final List usersWithLocalPathAccess;
>
> this.usersWithLocalPathAccess = Arrays.asList(
>
> conf.getTrimmedStrings(DFSConfigKeys.DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY));
> {code}
> In other word, the checking fails simply because the current user is not
> specified in the configuration setting of
> {{dfs.block.local-path-access.user}}. The log message should be much more
> clearer to make it easy for users to take actions, as demonstrated in the
> attached patch.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9782) RollingFileSystemSink should have configurable roll interval
[ https://issues.apache.org/jira/browse/HDFS-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HDFS-9782: --- Status: Open (was: Patch Available) The v8 patch looks good to me, but for the checkstyle nit - couple of lines are longer than 80 chars. +1, outside of that. Will commit this soon after that is fixed. > RollingFileSystemSink should have configurable roll interval > > > Key: HDFS-9782 > URL: https://issues.apache.org/jira/browse/HDFS-9782 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: HDFS-9782.001.patch, HDFS-9782.002.patch, > HDFS-9782.003.patch, HDFS-9782.004.patch, HDFS-9782.005.patch, > HDFS-9782.006.patch, HDFS-9782.007.patch, HDFS-9782.008.patch > > > Right now it defaults to rolling at the top of every hour. Instead that > interval should be configurable. The interval should also allow for some > play so that all hosts don't try to flush their files simultaneously. > I'm filing this in HDFS because I suspect it will involve touching the HDFS > tests. If it turns out not to, I'll move it into common instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-10417) Improve error message from checkBlockLocalPathAccess
[
https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee resolved HDFS-10417.
---
Resolution: Fixed
> Improve error message from checkBlockLocalPathAccess
>
>
> Key: HDFS-10417
> URL: https://issues.apache.org/jira/browse/HDFS-10417
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-10417.000.patch, HDFS-10417.001.patch,
> HDFS-10417.002.patch
>
>
> The exception msg thrown by {{checkBlockLocalPathAccess}} is very specific to
> the implementation detail. It's really hard for users to understand it unless
> she reads and understands the code.
> The code is shown as follows:
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private void checkBlockLocalPathAccess() throws IOException {
> checkKerberosAuthMethod("getBlockLocalPathInfo()");
> String currentUser =
> UserGroupInformation.getCurrentUser().getShortUserName();
> if (!usersWithLocalPathAccess.contains(currentUser)) {
> throw new AccessControlException(
> "Can't continue with getBlockLocalPathInfo() "
> + "authorization. The user " + currentUser
> + " is not allowed to call getBlockLocalPathInfo");
> }
> }
> {code}
> (basically she needs to understand the code logic of getBlockLocalPathInfo)
> \\
> Note that {{usersWithLocalPathAccess}} is a *private final* purely coming
> from the configuration settings of {{dfs.block.local-path-access.user}},
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private final List usersWithLocalPathAccess;
>
> this.usersWithLocalPathAccess = Arrays.asList(
>
> conf.getTrimmedStrings(DFSConfigKeys.DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY));
> {code}
> In other word, the checking fails simply because the current user is not
> specified in the configuration setting of
> {{dfs.block.local-path-access.user}}. The log message should be much more
> clearer to make it easy for users to take actions, as demonstrated in the
> attached patch.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10287) MiniDFSCluster should implement AutoCloseable
[
https://issues.apache.org/jira/browse/HDFS-10287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292190#comment-15292190
]
Andras Bokor commented on HDFS-10287:
-
[~ozawa] Can you help me on this? I could use this on one of the issues.
> MiniDFSCluster should implement AutoCloseable
> -
>
> Key: HDFS-10287
> URL: https://issues.apache.org/jira/browse/HDFS-10287
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Affects Versions: 2.7.0
>Reporter: John Zhuge
>Assignee: Andras Bokor
>Priority: Trivial
> Attachments: HDFS-10287.01.patch, HDFS-10287.02.patch
>
>
> {{MiniDFSCluster}} should implement {{AutoCloseable}} in order to support
> [try-with-resources|https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html].
> It will make test code a little cleaner and more reliable.
> Since {{AutoCloseable}} is only in Java 1.7 or later, this can not be
> backported to Hadoop version prior to 2.7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10417) Improve error message from checkBlockLocalPathAccess
[
https://issues.apache.org/jira/browse/HDFS-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292193#comment-15292193
]
Hudson commented on HDFS-10417:
---
SUCCESS: Integrated in Hadoop-trunk-Commit #9826 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9826/])
HDFS-10417. Improve error message from checkBlockLocalPathAccess. (kihwal: rev
204de430bf6a931e5688789c7a783c9c568a98c2)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitLocalRead.java
> Improve error message from checkBlockLocalPathAccess
>
>
> Key: HDFS-10417
> URL: https://issues.apache.org/jira/browse/HDFS-10417
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-10417.000.patch, HDFS-10417.001.patch,
> HDFS-10417.002.patch
>
>
> The exception msg thrown by {{checkBlockLocalPathAccess}} is very specific to
> the implementation detail. It's really hard for users to understand it unless
> she reads and understands the code.
> The code is shown as follows:
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private void checkBlockLocalPathAccess() throws IOException {
> checkKerberosAuthMethod("getBlockLocalPathInfo()");
> String currentUser =
> UserGroupInformation.getCurrentUser().getShortUserName();
> if (!usersWithLocalPathAccess.contains(currentUser)) {
> throw new AccessControlException(
> "Can't continue with getBlockLocalPathInfo() "
> + "authorization. The user " + currentUser
> + " is not allowed to call getBlockLocalPathInfo");
> }
> }
> {code}
> (basically she needs to understand the code logic of getBlockLocalPathInfo)
> \\
> Note that {{usersWithLocalPathAccess}} is a *private final* purely coming
> from the configuration settings of {{dfs.block.local-path-access.user}},
> {code:title=org.apache.hadoop.hdfs.server.datanode.DataNode|borderStyle=solid}
> private final List usersWithLocalPathAccess;
>
> this.usersWithLocalPathAccess = Arrays.asList(
>
> conf.getTrimmedStrings(DFSConfigKeys.DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY));
> {code}
> In other word, the checking fails simply because the current user is not
> specified in the configuration setting of
> {{dfs.block.local-path-access.user}}. The log message should be much more
> clearer to make it easy for users to take actions, as demonstrated in the
> attached patch.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9908) Datanode should tolerate disk scan failure during NN handshake
[
https://issues.apache.org/jira/browse/HDFS-9908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292208#comment-15292208
]
Wei-Chiu Chuang commented on HDFS-9908:
---
Interestingly, after HADOOP-12973, the exception thrown by du will be caught
and logged. That is to say, NN handshake will not be disrupt by this exception.
{code}
@Override
protected synchronized void refresh() {
if (duShell == null) {
duShell = new DUShell();
}
try {
duShell.startRefresh();
} catch (IOException ioe) {
LOG.warn("Could not get disk usage information", ioe);
}
}
{code}
Hiding a potential disk error in the log may not be the best option, IMO.
> Datanode should tolerate disk scan failure during NN handshake
> --
>
> Key: HDFS-9908
> URL: https://issues.apache.org/jira/browse/HDFS-9908
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
> Environment: CDH5.3.3
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HDFS-9908.001.patch, HDFS-9908.002.patch,
> HDFS-9908.003.patch, HDFS-9908.004.patch, HDFS-9908.005.patch,
> HDFS-9908.006.patch, HDFS-9908.007.patch
>
>
> DN may treat a disk scan failure exception as an NN handshake exception, and
> this can prevent a DN to join a cluster even if most of its disks are healthy.
> During NN handshake, DN initializes block pools. It will create a lock files
> per disk, and then scan the volumes. However, if the scanning throws
> exceptions due to disk failure, DN will think it's an exception because NN is
> inconsistent with the local storage (see {{DataNode#initBlockPool}}. As a
> result, it will attempt to reconnect to NN again.
> However, at this point, DN has not deleted its lock files on the disks. If it
> reconnects to NN again, it will think the same disks are already being used,
> and then it will fail handshake again because all disks can not be used (due
> to locking), and repeatedly. This will happen even if the DN has multiple
> disks, and only one of them fails. The DN will not be able to connect to NN
> despite just one failing disk. Note that it is possible to successfully
> create a lock file on a disk, and then has error scanning the disk.
> We saw this on a CDH 5.3.3 cluster (which is based on Apache Hadoop 2.5.0,
> and we still see the same bug in 3.0.0 trunk branch). The root cause is that
> DN treats an internal error (single disk failure) as an external one (NN
> handshake failure) and we should fix it.
> {code:title=DataNode.java}
> /**
>* One of the Block Pools has successfully connected to its NN.
>* This initializes the local storage for that block pool,
>* checks consistency of the NN's cluster ID, etc.
>*
>* If this is the first block pool to register, this also initializes
>* the datanode-scoped storage.
>*
>* @param bpos Block pool offer service
>* @throws IOException if the NN is inconsistent with the local storage.
>*/
> void initBlockPool(BPOfferService bpos) throws IOException {
> NamespaceInfo nsInfo = bpos.getNamespaceInfo();
> if (nsInfo == null) {
> throw new IOException("NamespaceInfo not found: Block pool " + bpos
> + " should have retrieved namespace info before initBlockPool.");
> }
>
> setClusterId(nsInfo.clusterID, nsInfo.getBlockPoolID());
> // Register the new block pool with the BP manager.
> blockPoolManager.addBlockPool(bpos);
>
> // In the case that this is the first block pool to connect, initialize
> // the dataset, block scanners, etc.
> initStorage(nsInfo);
> // Exclude failed disks before initializing the block pools to avoid
> startup
> // failures.
> checkDiskError();
> data.addBlockPool(nsInfo.getBlockPoolID(), conf); <- this line
> throws disk error exception
> blockScanner.enableBlockPoolId(bpos.getBlockPoolId());
> initDirectoryScanner(conf);
> }
> {code}
> {{FsVolumeList#addBlockPool}} is the source of exception.
> {code:title=FsVolumeList.java}
> void addBlockPool(final String bpid, final Configuration conf) throws
> IOException {
> long totalStartTime = Time.monotonicNow();
>
> final List exceptions = Collections.synchronizedList(
> new ArrayList());
> List blockPoolAddingThreads = new ArrayList();
> for (final FsVolumeImpl v : volumes) {
> Thread t = new Thread() {
> public void run() {
> try (FsVolumeReference ref = v.obtainReference()) {
> FsDatasetImpl.LOG.info("Scanning block pool " + bpid +
> " on volume " + v + "...");
> long startTime = Time.monotonicNow();
> v.addBlockPool(bpid, conf);
> long timeTaken
[jira] [Commented] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[
https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292215#comment-15292215
]
Hadoop QA commented on HDFS-10420:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 3m 41s
{color} | {color:red} root in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 10s
{color} | {color:red} hadoop-hdfs in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 48s
{color} | {color:red} branch/hadoop-hdfs-project/hadoop-hdfs no findbugs output
file (hadoop-hdfs-project/hadoop-hdfs/target/findbugsXml.xml) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 40s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 52s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91 with JDK v1.8.0_91
generated 4 new + 66 unchanged - 4 fixed = 70 total (was 70) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 9s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 9s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 3 new +
3 unchanged - 0 fixed = 6 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 18 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 10s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 6m 6s
{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101 with JDK
v1.7.0_101 generated 749 new + 7 unchanged - 7 fixed = 756 total (was 14)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 0s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 41s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color}
[jira] [Commented] (HDFS-9923) Datanode disk failure handling is not consistent
[
https://issues.apache.org/jira/browse/HDFS-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292228#comment-15292228
]
Wei-Chiu Chuang commented on HDFS-9923:
---
HADOOP-12973 unintentionally changed this behavior: du errors will be caught
and logged, but not propagated to the caller.
> Datanode disk failure handling is not consistent
>
>
> Key: HDFS-9923
> URL: https://issues.apache.org/jira/browse/HDFS-9923
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Wei-Chiu Chuang
> Labels: failure-handling, supportability
>
> Disk failures are hard to handle. This JIRA is created to discuss/improve
> disk failure handling in a better/consistent manner.
> For one thing, disks can fail in multiple different ways: the hardware might
> be failing, disk space is full, checksum error ... For others, hardware
> abstracts out the details, so it's hard for software to handle them.
> There are currently three disk check mechanisms in HDFS, as far as I know:
> {{BlockScanner}}, {{BlockPoolSlice#checkDirs}} and {{DU}}. Disk errors are
> handled differently.
> This JIRA is more focused on {{DU}} error handling. {{DU}} may emit errors
> like this:
> {noformat}
> 2016-02-18 02:23:36,224 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Caught
> exception while scanning /data/8/dfs/dn/current.
> Will throw later.
> ExitCodeException exitCode=1: du: cannot access
> `/data/8/dfs/dn/current/BP-1018136951-49.4.167.110-1403564146510/current/finalized/subdir228/subdir11/blk_
> 1088686909': Input/output error
> du: cannot access
> `/data/8/dfs/dn/current/BP-1018136951-49.4.167.110-1403564146510/current/finalized/subdir228/subdir11/blk_1088686909_14954023.meta':
> Inp
> ut/output error
> {noformat}
> I found {{DU}} errors are not handled consistently while working on HDFS-9908
> (Datanode should tolerate disk scan failure during NN handshake), and it all
> depends on who catches the exception.
> For example,
> * if DU returns error during NN handshake, DN will not be able to join the
> cluster at all(HDFS-9908);
> * however, if the same exception is caught in {{BlockPoolSlice#saveDfsUsed}},
> data node will only log a warning and do nothing (HDFS-5498).
> * in some cases, the exception handler invokes {{BlockPoolSlice#checkDirs}},
> but since it only checks three directories, it is very unlikely to find the
> files that have the error. {{BlockReceiver#(constructor)}}
> So my ask is: should the error be handled in a consistent manner? Should data
> node report to the name nodes about the disk failures (this is the
> BlockScanner approach), and should data node takes this volume offline
> automatically if DU returns an error? (this is the {{checkDirs}} approach)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10390) Implement asynchronous setAcl/getAclStatus for DistributedFileSystem
[
https://issues.apache.org/jira/browse/HDFS-10390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292235#comment-15292235
]
Hadoop QA commented on HDFS-10390:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
8s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 54s
{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 42s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 51s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestFsDatasetCache |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805040/HDFS-10390-HDFS-9924.006.patch
|
| JIRA Issue | HDFS-10390 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f59d1a38e9f5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / feb90ff |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15501/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HDFS-Build/15501/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15501/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-c
[jira] [Updated] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[ https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10420: - Attachment: HDFS-10420-HDFS-7240.03.patch Reattaching a patch, although I suspect the problems are due to Jenkins using JDK7. > Fix Ozone unit tests to use MiniOzoneCluster > > > Key: HDFS-10420 > URL: https://issues.apache.org/jira/browse/HDFS-10420 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10420-HDFS-7240.01.patch, > HDFS-10420-HDFS-7240.02.patch, HDFS-10420-HDFS-7240.03.patch > > > [~cnauroth] added MiniOzoneCluster support to the Ozone feature branch. We > should update all the Ozone tests to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[
https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292244#comment-15292244
]
Hadoop QA commented on HDFS-10420:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
22s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 27s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 2s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91 with JDK v1.8.0_91
generated 2 new + 33 unchanged - 2 fixed = 35 total (was 35) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 44s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101 with JDK v1.7.0_101
generated 2 new + 35 unchanged - 2 fixed = 37 total (was 37) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new +
1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 28s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 6s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 33s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 163m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_91 Failed junit tests | hadoo
[jira] [Commented] (HDFS-10390) Implement asynchronous setAcl/getAclStatus for DistributedFileSystem
[
https://issues.apache.org/jira/browse/HDFS-10390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292269#comment-15292269
]
Lei (Eddy) Xu commented on HDFS-10390:
--
Hi, [~xiaobingo]. Nice work.
In {{TestAsyncDFSRename#tesetAsyncAPIWithException}}, could you add {{fail()}}
to each {{try...catch}} to make sure an Exception was indeed thrown. i.e.,
{code}
try {
retFuture = ...
retFuture.get();
fail("");
} catch (ExcutionException ..) {
...
}
{code}
* Maybe we can move many of the tests in {{TestAsyncDFSRename}} to
{{TestAsyncDFS}}, as many of them are not {{rename}}-related. But it is not
necessary to be done in this JIRA.
+1 pending.
> Implement asynchronous setAcl/getAclStatus for DistributedFileSystem
>
>
> Key: HDFS-10390
> URL: https://issues.apache.org/jira/browse/HDFS-10390
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10390-HDFS-9924.000.patch,
> HDFS-10390-HDFS-9924.001.patch, HDFS-10390-HDFS-9924.002.patch,
> HDFS-10390-HDFS-9924.003.patch, HDFS-10390-HDFS-9924.004.patch,
> HDFS-10390-HDFS-9924.005.patch, HDFS-10390-HDFS-9924.006.patch
>
>
> This is proposed to implement asynchronous setAcl/getAclStatus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10424) DatanodeLifelineProtocol not able to use under security cluster
[
https://issues.apache.org/jira/browse/HDFS-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-10424:
-
Attachment: HDFS-10424.001.patch
[~gu chi], thank you for the bug report. I was able to reproduce it in a
secured cluster, and then I verified that the attached patch fixes it.
This patch also includes a new test suite designed to catch similar kinds of
bugs in the future. It works by scanning the list of protocol classes covered
by {{HDFSPolicyProvider}} and then comparing that to *Protocol interfaces
implemented by known RPC server classes. If it finds a protocol interface
implemented by a server, but not covered in the policy, then it fails. This
way, if we add new protocols, but forget to update {{HDFSPolicyProvider}}, then
the test will fail during pre-commit.
Interestingly, this test immediately exposed another potential offender:
{{ReconfigurationProtocol}}. I've coded the test to skip checking that one for
now in the interest of expediting the patch here. I'll file a separate JIRA
for follow-up on that one and contact contributors who have worked on
reconfiguration.
> DatanodeLifelineProtocol not able to use under security cluster
> ---
>
> Key: HDFS-10424
> URL: https://issues.apache.org/jira/browse/HDFS-10424
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: gu-chi
>Priority: Blocker
> Attachments: HDFS-10424.001.patch
>
>
> {quote}
> protocol org.apache.hadoop.hdfs.server.protocol.DatanodeLifelineProtocol is
> unauthorized for user * (auth:KERBEROS) | Server.java:1979
> {quote}
> I am using security cluster authenticate with kerberos, as I checked the the
> code, if security auth enabled, because the DatanodeLifelineProtocol is not
> inside HDFSPolicyProvider, when authorize in ServiceAuthorizationManager,
> AuthorizationException will be thrown at line 96.
> Please point me out if I am wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10424) DatanodeLifelineProtocol not able to use under security cluster
[
https://issues.apache.org/jira/browse/HDFS-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-10424:
-
Status: Patch Available (was: Open)
> DatanodeLifelineProtocol not able to use under security cluster
> ---
>
> Key: HDFS-10424
> URL: https://issues.apache.org/jira/browse/HDFS-10424
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: gu-chi
>Priority: Blocker
> Attachments: HDFS-10424.001.patch
>
>
> {quote}
> protocol org.apache.hadoop.hdfs.server.protocol.DatanodeLifelineProtocol is
> unauthorized for user * (auth:KERBEROS) | Server.java:1979
> {quote}
> I am using security cluster authenticate with kerberos, as I checked the the
> code, if security auth enabled, because the DatanodeLifelineProtocol is not
> inside HDFSPolicyProvider, when authorize in ServiceAuthorizationManager,
> AuthorizationException will be thrown at line 96.
> Please point me out if I am wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10424) DatanodeLifelineProtocol not able to use under security cluster
[
https://issues.apache.org/jira/browse/HDFS-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth reassigned HDFS-10424:
Assignee: Chris Nauroth
> DatanodeLifelineProtocol not able to use under security cluster
> ---
>
> Key: HDFS-10424
> URL: https://issues.apache.org/jira/browse/HDFS-10424
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: gu-chi
>Assignee: Chris Nauroth
>Priority: Blocker
> Attachments: HDFS-10424.001.patch
>
>
> {quote}
> protocol org.apache.hadoop.hdfs.server.protocol.DatanodeLifelineProtocol is
> unauthorized for user * (auth:KERBEROS) | Server.java:1979
> {quote}
> I am using security cluster authenticate with kerberos, as I checked the the
> code, if security auth enabled, because the DatanodeLifelineProtocol is not
> inside HDFSPolicyProvider, when authorize in ServiceAuthorizationManager,
> AuthorizationException will be thrown at line 96.
> Please point me out if I am wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10437) ReconfigurationProtocol not covered by HDFSPolicyProvider.
Chris Nauroth created HDFS-10437:
Summary: ReconfigurationProtocol not covered by HDFSPolicyProvider.
Key: HDFS-10437
URL: https://issues.apache.org/jira/browse/HDFS-10437
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.8.0
Reporter: Chris Nauroth
The {{HDFSPolicyProvider}} class contains an entry for defining the security
policy of each HDFS RPC protocol interface. {{ReconfigurationProtocol}} is not
listed currently. This may indicate that reconfiguration functionality is not
working correctly in secured clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10437) ReconfigurationProtocol not covered by HDFSPolicyProvider.
[
https://issues.apache.org/jira/browse/HDFS-10437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292285#comment-15292285
]
Chris Nauroth commented on HDFS-10437:
--
This came out of investigation on HDFS-10424, which reported a similar problem
for {{DatanodeLifelineProtocol}}. The test I wrote for the patch attached
there also indicated a possible problem for {{ReconfigurationProtocol}}.
[~xiaobingo], would you please check? Cc [~arpitagarwal].
> ReconfigurationProtocol not covered by HDFSPolicyProvider.
> --
>
> Key: HDFS-10437
> URL: https://issues.apache.org/jira/browse/HDFS-10437
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Chris Nauroth
>
> The {{HDFSPolicyProvider}} class contains an entry for defining the security
> policy of each HDFS RPC protocol interface. {{ReconfigurationProtocol}} is
> not listed currently. This may indicate that reconfiguration functionality
> is not working correctly in secured clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10415) TestDistributedFileSystem#testDFSCloseOrdering() fails on branch-2
[
https://issues.apache.org/jira/browse/HDFS-10415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292289#comment-15292289
]
Sangjin Lee commented on HDFS-10415:
#1 sounds like the right thing to do. [~andrew.wang], was there a reason that
{{MyDistributedFileSystem#delete()}} was left out in branch-2?
> TestDistributedFileSystem#testDFSCloseOrdering() fails on branch-2
> --
>
> Key: HDFS-10415
> URL: https://issues.apache.org/jira/browse/HDFS-10415
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.9.0
> Environment: jenkins
>Reporter: Sangjin Lee
>Assignee: Mingliang Liu
> Attachments: HDFS-10415-branch-2.000.patch,
> HDFS-10415-branch-2.001.patch, HDFS-10415.000.patch
>
>
> {noformat}
> Tests run: 24, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 51.096 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestDistributedFileSystem
> testDFSCloseOrdering(org.apache.hadoop.hdfs.TestDistributedFileSystem) Time
> elapsed: 0.045 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:790)
> at
> org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1417)
> at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2084)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1187)
> at
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testDFSCloseOrdering(TestDistributedFileSystem.java:217)
> {noformat}
> This is with Java 8 on Mac. It passes fine on trunk. I haven't tried other
> combinations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10437) ReconfigurationProtocol not covered by HDFSPolicyProvider.
[
https://issues.apache.org/jira/browse/HDFS-10437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292293#comment-15292293
]
Arpit Agarwal commented on HDFS-10437:
--
Nice catch [~cnauroth], thank you for the heads up!
> ReconfigurationProtocol not covered by HDFSPolicyProvider.
> --
>
> Key: HDFS-10437
> URL: https://issues.apache.org/jira/browse/HDFS-10437
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Chris Nauroth
>
> The {{HDFSPolicyProvider}} class contains an entry for defining the security
> policy of each HDFS RPC protocol interface. {{ReconfigurationProtocol}} is
> not listed currently. This may indicate that reconfiguration functionality
> is not working correctly in secured clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10437) ReconfigurationProtocol not covered by HDFSPolicyProvider.
[
https://issues.apache.org/jira/browse/HDFS-10437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292294#comment-15292294
]
Arpit Agarwal commented on HDFS-10437:
--
cc [~xiaobingo]
> ReconfigurationProtocol not covered by HDFSPolicyProvider.
> --
>
> Key: HDFS-10437
> URL: https://issues.apache.org/jira/browse/HDFS-10437
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Chris Nauroth
>
> The {{HDFSPolicyProvider}} class contains an entry for defining the security
> policy of each HDFS RPC protocol interface. {{ReconfigurationProtocol}} is
> not listed currently. This may indicate that reconfiguration functionality
> is not working correctly in secured clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7766) Add a flag to WebHDFS op=CREATE to not respond with a 307 redirect
[
https://issues.apache.org/jira/browse/HDFS-7766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated HDFS-7766:
---
Attachment: HDFS-7766.06.patch
Here's a patch with some improvements to the documentation. I would really like
to get this into the 2.8.0 release. If we can get over this hump, we would be
able to include file uploading and preview functionality into 2.8.0.
> Add a flag to WebHDFS op=CREATE to not respond with a 307 redirect
> --
>
> Key: HDFS-7766
> URL: https://issues.apache.org/jira/browse/HDFS-7766
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HDFS-7766.01.patch, HDFS-7766.02.patch,
> HDFS-7766.03.patch, HDFS-7766.04.patch, HDFS-7766.04.patch,
> HDFS-7766.05.patch, HDFS-7766.06.patch
>
>
> Please see
> https://issues.apache.org/jira/browse/HDFS-7588?focusedCommentId=14276192&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14276192
> A backwards compatible manner we can fix this is to add a flag on the request
> which would disable the redirect, i.e.
> {noformat}
> curl -i -X PUT
> "http://:/webhdfs/v1/?op=CREATE[&noredirect=]
> {noformat}
> returns 200 with the DN location in the response.
> This would allow the Browser clients to get the redirect URL to put the file
> to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7240) Object store in HDFS
[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292351#comment-15292351 ] Zhe Zhang commented on HDFS-7240: - Thanks for the discussions [~andrew.wang] [~anu]. I'm still trying to catch up all the new updates (looking forward to the updated design doc, maybe also post the ApacheCon video?). Meanwhile, some thoughts around EC: bq. My biggest concern is that erasure coding is not a first-class consideration in this system, and seems like it will be quite difficult to implement. I agree. The most difficult parts in building HDFS-EC was to remove / generalize inflexible assumptions in HDFS block management and namespace logics about replicas etc. So it would help a lot to at least conceptually discuss the plan for EC implementation in the new design doc. Another question about reading the ApacheCon slides: the question "Why an Object Store" was well answered. How about "why an object store as part of HDFS"? IIUC Ozone is only leveraging a very small portion of HDFS code. Why should it be a part of HDFS instead of a separate project? > Object store in HDFS > > > Key: HDFS-7240 > URL: https://issues.apache.org/jira/browse/HDFS-7240 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: Ozone-architecture-v1.pdf, ozone_user_v0.pdf > > > This jira proposes to add object store capabilities into HDFS. > As part of the federation work (HDFS-1052) we separated block storage as a > generic storage layer. Using the Block Pool abstraction, new kinds of > namespaces can be built on top of the storage layer i.e. datanodes. > In this jira I will explore building an object store using the datanode > storage, but independent of namespace metadata. > I will soon update with a detailed design document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10438) When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.
Chris Nauroth created HDFS-10438:
Summary: When NameNode HA is configured to use the lifeline RPC
server, it should log the address of that server.
Key: HDFS-10438
URL: https://issues.apache.org/jira/browse/HDFS-10438
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode
Reporter: KWON BYUNGCHANG
Assignee: Chris Nauroth
Priority: Minor
As reported by [~magnum]:
I have configured below
{code}
dfs.namenode.servicerpc-address.xdev.nn1=my.host.com:8040
dfs.namenode.lifeline.rpc-address.xdev.nn1=my.host.com:8041
{code}
servicerpc port is 8040, lifeline port is 8041.
however zkfc daemon is logging using servicerpc port.
It may cause confusion.
thank you.
{code}
2016-05-19 19:18:40,566 WARN ha.HealthMonitor
(HealthMonitor.java:doHealthChecks(207)) - Service health check failed for
NameNode at my.host.com/10.114.87.91:8040: The NameNode has no resources
available
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9466) TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky
[
https://issues.apache.org/jira/browse/HDFS-9466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292366#comment-15292366
]
Wei-Chiu Chuang commented on HDFS-9466:
---
Hello [~cmccabe], would you mind to review it again? Thank you very much!
> TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky
>
>
> Key: HDFS-9466
> URL: https://issues.apache.org/jira/browse/HDFS-9466
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, hdfs-client
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HDFS-9466.001.patch, HDFS-9466.002.patch,
> org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache-output.txt
>
>
> This test is flaky and fails quite frequently in trunk.
> Error Message
> expected:<1> but was:<2>
> Stacktrace
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache$17.accept(TestShortCircuitCache.java:636)
> at
> org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.visit(ShortCircuitRegistry.java:395)
> at
> org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache.checkNumberOfSegmentsAndSlots(TestShortCircuitCache.java:631)
> at
> org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache.testDataXceiverCleansUpSlotsOnFailure(TestShortCircuitCache.java:684)
> {noformat}
> Thanks to [~xiaochen] for identifying the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10438) When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.
[
https://issues.apache.org/jira/browse/HDFS-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-10438:
-
Status: Patch Available (was: Open)
> When NameNode HA is configured to use the lifeline RPC server, it should log
> the address of that server.
>
>
> Key: HDFS-10438
> URL: https://issues.apache.org/jira/browse/HDFS-10438
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Reporter: KWON BYUNGCHANG
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-10438.001.patch
>
>
> As reported by [~magnum]:
> I have configured below
> {code}
> dfs.namenode.servicerpc-address.xdev.nn1=my.host.com:8040
> dfs.namenode.lifeline.rpc-address.xdev.nn1=my.host.com:8041
> {code}
> servicerpc port is 8040, lifeline port is 8041.
> however zkfc daemon is logging using servicerpc port.
> It may cause confusion.
> thank you.
> {code}
> 2016-05-19 19:18:40,566 WARN ha.HealthMonitor
> (HealthMonitor.java:doHealthChecks(207)) - Service health check failed for
> NameNode at my.host.com/10.114.87.91:8040: The NameNode has no resources
> available
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9311) Support optional offload of NameNode HA service health checks to a separate RPC server.
[ https://issues.apache.org/jira/browse/HDFS-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292373#comment-15292373 ] Chris Nauroth commented on HDFS-9311: - [~magnum], thank you for your comment. I think we can improve that log message to use the lifeline address when it's configured. I filed a follow-up patch on HDFS-10438. > Support optional offload of NameNode HA service health checks to a separate > RPC server. > --- > > Key: HDFS-9311 > URL: https://issues.apache.org/jira/browse/HDFS-9311 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, namenode >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 2.8.0 > > Attachments: HDFS-9311.001.patch, HDFS-9311.002.patch, > HDFS-9311.003.patch > > > When a NameNode is overwhelmed with load, it can lead to resource exhaustion > of the RPC handler pools (both client-facing and service-facing). > Eventually, this blocks the health check RPC issued from ZKFC, which triggers > a failover. Depending on fencing configuration, the former active NameNode > may be killed. In an overloaded situation, the new active NameNode is likely > to suffer the same fate, because client load patterns don't change after the > failover. This can degenerate into flapping between the 2 NameNodes without > real recovery. If a NameNode had been killed by fencing, then it would have > to transition through safe mode, further delaying time to recovery. > This issue proposes a separate, optional RPC server at the NameNode for > isolating the HA health checks. These health checks are lightweight > operations that do not suffer from contention issues on the namesystem lock > or other shared resources. Isolating the RPC handlers is sufficient to avoid > this situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[
https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292385#comment-15292385
]
Hadoop QA commented on HDFS-10420:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
8s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 57s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs generated 2 new + 114 unchanged -
2 fixed = 116 total (was 116) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 29s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 18 new +
38 unchanged - 6 fixed = 56 total (was 44) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 25s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 0s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.TestAsyncDFSRename |
| | hadoop.hdfs.server.datanode.TestBlockPoolManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805067/HDFS-10420-HDFS-7240.03.patch
|
| JIRA Issue | HDFS-10420 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4863f52be5ed 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 76279f5 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| javac | hadoop-hdfs-project_hadoop-hdfs:
https://builds.apache.org/job/PreCommit-HDFS-Build/15502/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/15502/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15502/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
https://builds.ap
[jira] [Updated] (HDFS-10438) When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.
[
https://issues.apache.org/jira/browse/HDFS-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-10438:
-
Attachment: HDFS-10438.001.patch
I'm attaching a patch that uses the lifeline address in the string that gets
logged if a lifeline address is configured. I also updated a test to check for
the correct string representation. While working on this, I noticed that
{{MiniDFSCluster}} was not quite doing the right thing for copying the lifeline
address configuration key in a multi-nameservice/multi-namenode setup.
> When NameNode HA is configured to use the lifeline RPC server, it should log
> the address of that server.
>
>
> Key: HDFS-10438
> URL: https://issues.apache.org/jira/browse/HDFS-10438
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Reporter: KWON BYUNGCHANG
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-10438.001.patch
>
>
> As reported by [~magnum]:
> I have configured below
> {code}
> dfs.namenode.servicerpc-address.xdev.nn1=my.host.com:8040
> dfs.namenode.lifeline.rpc-address.xdev.nn1=my.host.com:8041
> {code}
> servicerpc port is 8040, lifeline port is 8041.
> however zkfc daemon is logging using servicerpc port.
> It may cause confusion.
> thank you.
> {code}
> 2016-05-19 19:18:40,566 WARN ha.HealthMonitor
> (HealthMonitor.java:doHealthChecks(207)) - Service health check failed for
> NameNode at my.host.com/10.114.87.91:8040: The NameNode has no resources
> available
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10438) When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.
[
https://issues.apache.org/jira/browse/HDFS-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292386#comment-15292386
]
Chris Nauroth commented on HDFS-10438:
--
Just in case it's not obvious, the {{toString}} method I'm changing gets called
from {{org.apache.hadoop.ha.HealthMonitor}} up in hadoop-common. That's why
changing the {{toString}} code fixes the log message.
> When NameNode HA is configured to use the lifeline RPC server, it should log
> the address of that server.
>
>
> Key: HDFS-10438
> URL: https://issues.apache.org/jira/browse/HDFS-10438
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Reporter: KWON BYUNGCHANG
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-10438.001.patch
>
>
> As reported by [~magnum]:
> I have configured below
> {code}
> dfs.namenode.servicerpc-address.xdev.nn1=my.host.com:8040
> dfs.namenode.lifeline.rpc-address.xdev.nn1=my.host.com:8041
> {code}
> servicerpc port is 8040, lifeline port is 8041.
> however zkfc daemon is logging using servicerpc port.
> It may cause confusion.
> thank you.
> {code}
> 2016-05-19 19:18:40,566 WARN ha.HealthMonitor
> (HealthMonitor.java:doHealthChecks(207)) - Service health check failed for
> NameNode at my.host.com/10.114.87.91:8040: The NameNode has no resources
> available
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10437) ReconfigurationProtocol not covered by HDFSPolicyProvider.
[
https://issues.apache.org/jira/browse/HDFS-10437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292394#comment-15292394
]
Xiaobing Zhou commented on HDFS-10437:
--
Thanks [~cnauroth] for the catch. We should add ReconfigurationProtocol to
HDFSPolicyProvider.
> ReconfigurationProtocol not covered by HDFSPolicyProvider.
> --
>
> Key: HDFS-10437
> URL: https://issues.apache.org/jira/browse/HDFS-10437
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Chris Nauroth
>
> The {{HDFSPolicyProvider}} class contains an entry for defining the security
> policy of each HDFS RPC protocol interface. {{ReconfigurationProtocol}} is
> not listed currently. This may indicate that reconfiguration functionality
> is not working correctly in secured clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10437) ReconfigurationProtocol not covered by HDFSPolicyProvider.
[
https://issues.apache.org/jira/browse/HDFS-10437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292395#comment-15292395
]
Chris Nauroth commented on HDFS-10437:
--
[~xiaobingo], thanks for confirmation. When we do that, let's also update the
new {{TestHDFSPolicyProvider}} suite that I'm adding in HDFS-10424 to remove
the special-case check for {{ReconfigurationProtocol}}.
> ReconfigurationProtocol not covered by HDFSPolicyProvider.
> --
>
> Key: HDFS-10437
> URL: https://issues.apache.org/jira/browse/HDFS-10437
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Chris Nauroth
>
> The {{HDFSPolicyProvider}} class contains an entry for defining the security
> policy of each HDFS RPC protocol interface. {{ReconfigurationProtocol}} is
> not listed currently. This may indicate that reconfiguration functionality
> is not working correctly in secured clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10437) ReconfigurationProtocol not covered by HDFSPolicyProvider.
[
https://issues.apache.org/jira/browse/HDFS-10437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou reassigned HDFS-10437:
Assignee: Xiaobing Zhou
> ReconfigurationProtocol not covered by HDFSPolicyProvider.
> --
>
> Key: HDFS-10437
> URL: https://issues.apache.org/jira/browse/HDFS-10437
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Chris Nauroth
>Assignee: Xiaobing Zhou
>
> The {{HDFSPolicyProvider}} class contains an entry for defining the security
> policy of each HDFS RPC protocol interface. {{ReconfigurationProtocol}} is
> not listed currently. This may indicate that reconfiguration functionality
> is not working correctly in secured clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10437) ReconfigurationProtocol not covered by HDFSPolicyProvider.
[
https://issues.apache.org/jira/browse/HDFS-10437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292401#comment-15292401
]
Xiaobing Zhou commented on HDFS-10437:
--
Ok, I will jump to this after HDFS-10424 is committed. Thank you.
> ReconfigurationProtocol not covered by HDFSPolicyProvider.
> --
>
> Key: HDFS-10437
> URL: https://issues.apache.org/jira/browse/HDFS-10437
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Chris Nauroth
>Assignee: Xiaobing Zhou
>
> The {{HDFSPolicyProvider}} class contains an entry for defining the security
> policy of each HDFS RPC protocol interface. {{ReconfigurationProtocol}} is
> not listed currently. This may indicate that reconfiguration functionality
> is not working correctly in secured clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10408) Add tests for out-of-order asynchronous rename/setPermission/setOwner
[ https://issues.apache.org/jira/browse/HDFS-10408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou reassigned HDFS-10408: Assignee: Xiaobing Zhou > Add tests for out-of-order asynchronous rename/setPermission/setOwner > - > > Key: HDFS-10408 > URL: https://issues.apache.org/jira/browse/HDFS-10408 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10408-HDFS-9924.000.patch > > > HDFS-10224 and HDFS-10346 mostly test the batch style async request/response. > Out-of-order case should also be tested. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10413) Implement asynchronous listStatus for DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou reassigned HDFS-10413: Assignee: Xiaobing Zhou > Implement asynchronous listStatus for DistributedFileSystem > --- > > Key: HDFS-10413 > URL: https://issues.apache.org/jira/browse/HDFS-10413 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > Per the > [comment|https://issues.apache.org/jira/browse/HDFS-9924?focusedCommentId=15285597&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15285597] > from [~mingma], this Jira tracks efforts of implementing async listStatus. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10430) Refactor FileSystem#checkAccessPermissions for better reuse from tests
[ https://issues.apache.org/jira/browse/HDFS-10430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou reassigned HDFS-10430: Assignee: Xiaobing Zhou > Refactor FileSystem#checkAccessPermissions for better reuse from tests > -- > > Key: HDFS-10430 > URL: https://issues.apache.org/jira/browse/HDFS-10430 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > FileSystem#checkAccessPermissions could be used in a bunch of tests from > different projects, but it's in hadoop-common, which is not visible in some > cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10431) Refactor Async DFS related tests to reuse shared instance of AsyncDistributedFileSystem instance to speed up tests
[ https://issues.apache.org/jira/browse/HDFS-10431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou reassigned HDFS-10431: Assignee: Xiaobing Zhou > Refactor Async DFS related tests to reuse shared instance of > AsyncDistributedFileSystem instance to speed up tests > -- > > Key: HDFS-10431 > URL: https://issues.apache.org/jira/browse/HDFS-10431 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > Limit of max async calls(i.e. ipc.client.async.calls.max) is set and cached > in ipc.Client. Client instances are cached based on SocketFactory. In order > to test different cases in various limits, every test (e.g. > TestAsyncDFSRename and TestAsyncDFS) creates separate instance of > MiniDFSCluster and that of AsyncDistributedFileSystem hence. This is not > efficient in that tests may take long time to bootstrap MiniDFSClusters. It's > even worse if cluster needs to restart in the middle. This proposes to do > refactoring to use shared instance of AsyncDistributedFileSystem for speedup. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10433) Make retry also works well for Async DFS
[ https://issues.apache.org/jira/browse/HDFS-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10433: - Assignee: Tsz Wo Nicholas Sze > Make retry also works well for Async DFS > > > Key: HDFS-10433 > URL: https://issues.apache.org/jira/browse/HDFS-10433 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Xiaobing Zhou >Assignee: Tsz Wo Nicholas Sze > > In current Async DFS implementation, file system calls are invoked and > returns Future immediately to clients. Clients call Future#get to retrieve > final results. Future#get internally invokes a chain of callbacks residing in > ClientNamenodeProtocolTranslatorPB, ProtobufRpcEngine and ipc.Client. The > callback path bypasses the original retry layer/logic designed for > synchronous DFS. This proposes refactoring to make retry also works for Async > DFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10424) DatanodeLifelineProtocol not able to use under security cluster
[
https://issues.apache.org/jira/browse/HDFS-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292487#comment-15292487
]
Hadoop QA commented on HDFS-10424:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
0s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 10s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 42s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 57s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 6s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 20s
{color} | {color:red} root: patch generated 3 new + 156 unchanged - 0 fixed =
159 total (was 156) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 31s {color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 28s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 128m 20s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.TestFileAppend |
| Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805069/HDFS-10424.001.patch |
| JIRA Issue | HDFS-10424 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c45856748611 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 42c22f7 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/15503/artifact/patchprocess/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15503/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15503/artifact/pa
[jira] [Commented] (HDFS-7766) Add a flag to WebHDFS op=CREATE to not respond with a 307 redirect
[
https://issues.apache.org/jira/browse/HDFS-7766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292499#comment-15292499
]
Hadoop QA commented on HDFS-7766:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 35s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 15s
{color} | {color:green} hadoop-hdfs-project generated 0 new + 52 unchanged - 1
fixed = 52 total (was 53) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 38s
{color} | {color:green} hadoop-hdfs-project in the patch passed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 32s
{color} | {color:red} hadoop-hdfs-project: patch generated 20 new + 160
unchanged - 15 fixed = 180 total (was 175) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 53s
{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 49s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 36s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805077/HDFS-7766.06.patch |
| JIRA Issue | HDFS-7766 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 260b32dcffdf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 42c22f7 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/15504/artifact/patchprocess/diff-checkstyl
[jira] [Commented] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[
https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292518#comment-15292518
]
Anu Engineer commented on HDFS-10420:
-
[~arpitagarwal] thank you for cleaning up all ozone tests. I am so glad to see
that ozone tests are now using MiniOzoneCluster instead of MiniDFSCluster. This
was a work item in my list. I appreciate you taking care of that.
Overall the code looks great, one minor comment.
# {{OzoneClientUtils.java:116}} has a debug tag in the log message which you
might want to remove.
> Fix Ozone unit tests to use MiniOzoneCluster
>
>
> Key: HDFS-10420
> URL: https://issues.apache.org/jira/browse/HDFS-10420
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, test
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-10420-HDFS-7240.01.patch,
> HDFS-10420-HDFS-7240.02.patch, HDFS-10420-HDFS-7240.03.patch
>
>
> [~cnauroth] added MiniOzoneCluster support to the Ozone feature branch. We
> should update all the Ozone tests to use it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10438) When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.
[
https://issues.apache.org/jira/browse/HDFS-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292535#comment-15292535
]
Hadoop QA commented on HDFS-10438:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 31s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new +
208 unchanged - 0 fixed = 209 total (was 208) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 13s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 42s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestEditLog |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.TestAsyncDFSRename |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805078/HDFS-10438.001.patch |
| JIRA Issue | HDFS-10438 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 99dd3ce764e7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 42c22f7 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/15505/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15505/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HDFS-Build/15505/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15505/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15505/console |
[jira] [Assigned] (HDFS-10436) dfs.block.access.token.enable should default on when security is !simple
[ https://issues.apache.org/jira/browse/HDFS-10436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin reassigned HDFS-10436: Assignee: Yiqun Lin > dfs.block.access.token.enable should default on when security is !simple > > > Key: HDFS-10436 > URL: https://issues.apache.org/jira/browse/HDFS-10436 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Yiqun Lin > > Unless there is a valid configuration where dfs.block.access.token.enable is > off and security is on, then rather than shutdown we should just enable the > block access tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10436) dfs.block.access.token.enable should default on when security is !simple
[
https://issues.apache.org/jira/browse/HDFS-10436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292544#comment-15292544
]
Yiqun Lin commented on HDFS-10436:
--
HI,[~aw], thanks for reporting this. I agree with you. If the UGI security is
enabled and you forget to enable the {{dfs.block.access.token.enable}}, then
still use the default value(here is false). And it will cause the IOException.
Like these:
{code}
private static BlockTokenSecretManager createBlockTokenSecretManager(
final Configuration conf) throws IOException {
final boolean isEnabled = conf.getBoolean(
DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY,
DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_DEFAULT);
LOG.info(DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY + "=" + isEnabled);
if (!isEnabled) {
if (UserGroupInformation.isSecurityEnabled()) {
String errMessage = "Security is enabled but block access tokens " +
"(via " + DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY + ") " +
"aren't enabled. This may cause issues " +
"when clients attempt to connect to a DataNode. Aborting NameNode";
throw new IOException(errMessage);
}
return null;
}
{code}
In {{DataNode#checkSecureConfig}}, there is also a similar problem. Attach a
patch for this.
> dfs.block.access.token.enable should default on when security is !simple
>
>
> Key: HDFS-10436
> URL: https://issues.apache.org/jira/browse/HDFS-10436
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Yiqun Lin
>
> Unless there is a valid configuration where dfs.block.access.token.enable is
> off and security is on, then rather than shutdown we should just enable the
> block access tokens.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10436) dfs.block.access.token.enable should default on when security is !simple
[
https://issues.apache.org/jira/browse/HDFS-10436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292544#comment-15292544
]
Yiqun Lin edited comment on HDFS-10436 at 5/20/16 2:12 AM:
---
HI,[~aw], thanks for reporting this. I agree with you. If the UGI security is
enabled and you forget to enable the {{dfs.block.access.token.enable}}, then
still use the default value(here is false). And it will return null here. Like
these:
{code}
private static BlockTokenSecretManager createBlockTokenSecretManager(
final Configuration conf) throws IOException {
final boolean isEnabled = conf.getBoolean(
DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY,
DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_DEFAULT);
LOG.info(DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY + "=" + isEnabled);
if (!isEnabled) {
if (UserGroupInformation.isSecurityEnabled()) {
String errMessage = "Security is enabled but block access tokens " +
"(via " + DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY + ") " +
"aren't enabled. This may cause issues " +
"when clients attempt to connect to a DataNode. Aborting NameNode";
throw new IOException(errMessage);
}
return null;
}
{code}
In {{DataNode#checkSecureConfig}}, there is also a similar problem. Attach a
patch for this later.
was (Author: linyiqun):
HI,[~aw], thanks for reporting this. I agree with you. If the UGI security is
enabled and you forget to enable the {{dfs.block.access.token.enable}}, then
still use the default value(here is false). And it will cause the IOException.
Like these:
{code}
private static BlockTokenSecretManager createBlockTokenSecretManager(
final Configuration conf) throws IOException {
final boolean isEnabled = conf.getBoolean(
DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY,
DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_DEFAULT);
LOG.info(DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY + "=" + isEnabled);
if (!isEnabled) {
if (UserGroupInformation.isSecurityEnabled()) {
String errMessage = "Security is enabled but block access tokens " +
"(via " + DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY + ") " +
"aren't enabled. This may cause issues " +
"when clients attempt to connect to a DataNode. Aborting NameNode";
throw new IOException(errMessage);
}
return null;
}
{code}
In {{DataNode#checkSecureConfig}}, there is also a similar problem. Attach a
patch for this.
> dfs.block.access.token.enable should default on when security is !simple
>
>
> Key: HDFS-10436
> URL: https://issues.apache.org/jira/browse/HDFS-10436
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Yiqun Lin
>
> Unless there is a valid configuration where dfs.block.access.token.enable is
> off and security is on, then rather than shutdown we should just enable the
> block access tokens.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10436) dfs.block.access.token.enable should default on when security is !simple
[
https://issues.apache.org/jira/browse/HDFS-10436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292544#comment-15292544
]
Yiqun Lin edited comment on HDFS-10436 at 5/20/16 2:15 AM:
---
HI,[~aw], thanks for reporting this. I agree with you. If the UGI security is
enabled and you forget to enable the {{dfs.block.access.token.enable}}, then
still use the default value(here is false). And it will throw IOException here.
Like these:
{code}
private static BlockTokenSecretManager createBlockTokenSecretManager(
final Configuration conf) throws IOException {
final boolean isEnabled = conf.getBoolean(
DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY,
DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_DEFAULT);
LOG.info(DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY + "=" + isEnabled);
if (!isEnabled) {
if (UserGroupInformation.isSecurityEnabled()) {
String errMessage = "Security is enabled but block access tokens " +
"(via " + DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY + ") " +
"aren't enabled. This may cause issues " +
"when clients attempt to connect to a DataNode. Aborting NameNode";
throw new IOException(errMessage);
}
return null;
}
{code}
In {{DataNode#checkSecureConfig}}, there is also a similar problem. Attach a
patch for this later.
was (Author: linyiqun):
HI,[~aw], thanks for reporting this. I agree with you. If the UGI security is
enabled and you forget to enable the {{dfs.block.access.token.enable}}, then
still use the default value(here is false). And it will return null here. Like
these:
{code}
private static BlockTokenSecretManager createBlockTokenSecretManager(
final Configuration conf) throws IOException {
final boolean isEnabled = conf.getBoolean(
DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY,
DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_DEFAULT);
LOG.info(DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY + "=" + isEnabled);
if (!isEnabled) {
if (UserGroupInformation.isSecurityEnabled()) {
String errMessage = "Security is enabled but block access tokens " +
"(via " + DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY + ") " +
"aren't enabled. This may cause issues " +
"when clients attempt to connect to a DataNode. Aborting NameNode";
throw new IOException(errMessage);
}
return null;
}
{code}
In {{DataNode#checkSecureConfig}}, there is also a similar problem. Attach a
patch for this later.
> dfs.block.access.token.enable should default on when security is !simple
>
>
> Key: HDFS-10436
> URL: https://issues.apache.org/jira/browse/HDFS-10436
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Yiqun Lin
>
> Unless there is a valid configuration where dfs.block.access.token.enable is
> off and security is on, then rather than shutdown we should just enable the
> block access tokens.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10424) DatanodeLifelineProtocol not able to use under security cluster
[
https://issues.apache.org/jira/browse/HDFS-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292656#comment-15292656
]
Arpit Agarwal commented on HDFS-10424:
--
+1. Neat unit test [~cnauroth]. Thanks for taking care of this and for finding
the ReconfigurationProtocol issue.
> DatanodeLifelineProtocol not able to use under security cluster
> ---
>
> Key: HDFS-10424
> URL: https://issues.apache.org/jira/browse/HDFS-10424
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: gu-chi
>Assignee: Chris Nauroth
>Priority: Blocker
> Attachments: HDFS-10424.001.patch
>
>
> {quote}
> protocol org.apache.hadoop.hdfs.server.protocol.DatanodeLifelineProtocol is
> unauthorized for user * (auth:KERBEROS) | Server.java:1979
> {quote}
> I am using security cluster authenticate with kerberos, as I checked the the
> code, if security auth enabled, because the DatanodeLifelineProtocol is not
> inside HDFSPolicyProvider, when authorize in ServiceAuthorizationManager,
> AuthorizationException will be thrown at line 96.
> Please point me out if I am wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10436) dfs.block.access.token.enable should default on when security is !simple
[ https://issues.apache.org/jira/browse/HDFS-10436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-10436: - Attachment: HDFS-10436.001.patch > dfs.block.access.token.enable should default on when security is !simple > > > Key: HDFS-10436 > URL: https://issues.apache.org/jira/browse/HDFS-10436 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Yiqun Lin > Attachments: HDFS-10436.001.patch > > > Unless there is a valid configuration where dfs.block.access.token.enable is > off and security is on, then rather than shutdown we should just enable the > block access tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10436) dfs.block.access.token.enable should default on when security is !simple
[
https://issues.apache.org/jira/browse/HDFS-10436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10436:
-
Status: Patch Available (was: Open)
Attach a simple patch for this. I think it will not affect the main logic in
HDFS, the default value only used in {{DataNode#checkSecureConfig}} and
{{BlockManager#createBlockTokenSecretManager}}. Thanks for review!
> dfs.block.access.token.enable should default on when security is !simple
>
>
> Key: HDFS-10436
> URL: https://issues.apache.org/jira/browse/HDFS-10436
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Yiqun Lin
> Attachments: HDFS-10436.001.patch
>
>
> Unless there is a valid configuration where dfs.block.access.token.enable is
> off and security is on, then rather than shutdown we should just enable the
> block access tokens.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[ https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10420: - Attachment: HDFS-10420-HDFS-7240.04.patch Thanks for the review [~anu]. I removed the debug log statement and fixed a bunch of checkstyle issues. Reattaching patch for Jenkins run. > Fix Ozone unit tests to use MiniOzoneCluster > > > Key: HDFS-10420 > URL: https://issues.apache.org/jira/browse/HDFS-10420 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10420-HDFS-7240.01.patch, > HDFS-10420-HDFS-7240.02.patch, HDFS-10420-HDFS-7240.03.patch, > HDFS-10420-HDFS-7240.04.patch > > > [~cnauroth] added MiniOzoneCluster support to the Ozone feature branch. We > should update all the Ozone tests to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster
[ https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10420: - Attachment: HDFS-10420-HDFS-7240.05.patch Also fix a javac warning exposed by removing usage of raw types. > Fix Ozone unit tests to use MiniOzoneCluster > > > Key: HDFS-10420 > URL: https://issues.apache.org/jira/browse/HDFS-10420 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-10420-HDFS-7240.01.patch, > HDFS-10420-HDFS-7240.02.patch, HDFS-10420-HDFS-7240.03.patch, > HDFS-10420-HDFS-7240.04.patch, HDFS-10420-HDFS-7240.05.patch > > > [~cnauroth] added MiniOzoneCluster support to the Ozone feature branch. We > should update all the Ozone tests to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10438) When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.
[
https://issues.apache.org/jira/browse/HDFS-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292686#comment-15292686
]
Arpit Agarwal commented on HDFS-10438:
--
+1 thanks [~cnauroth].
> When NameNode HA is configured to use the lifeline RPC server, it should log
> the address of that server.
>
>
> Key: HDFS-10438
> URL: https://issues.apache.org/jira/browse/HDFS-10438
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Reporter: KWON BYUNGCHANG
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-10438.001.patch
>
>
> As reported by [~magnum]:
> I have configured below
> {code}
> dfs.namenode.servicerpc-address.xdev.nn1=my.host.com:8040
> dfs.namenode.lifeline.rpc-address.xdev.nn1=my.host.com:8041
> {code}
> servicerpc port is 8040, lifeline port is 8041.
> however zkfc daemon is logging using servicerpc port.
> It may cause confusion.
> thank you.
> {code}
> 2016-05-19 19:18:40,566 WARN ha.HealthMonitor
> (HealthMonitor.java:doHealthChecks(207)) - Service health check failed for
> NameNode at my.host.com/10.114.87.91:8040: The NameNode has no resources
> available
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10439) Update setOwner doc in HdfsPermissionsGuide
John Zhuge created HDFS-10439:
-
Summary: Update setOwner doc in HdfsPermissionsGuide
Key: HDFS-10439
URL: https://issues.apache.org/jira/browse/HDFS-10439
Project: Hadoop HDFS
Issue Type: Bug
Components: documentation
Affects Versions: 2.8.0
Reporter: John Zhuge
Assignee: John Zhuge
Priority: Minor
In {{HdfsPermissionsGuide}}:
bq. HDFS super-user access is not required to change the group, but the caller
must be a member of the specified group.
should be
bq. HDFS super-user access is not required to change the group, but the caller
must be the owner of the file and a member of the specified group.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
