[jira] [Updated] (HDFS-10446) Add interleaving tests for async DFS API

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10446:
-
Summary: Add interleaving tests for async DFS API  (was: Add interleaving 
tests for asynchronous async DFS API)

> Add interleaving tests for async DFS API
> 
>
> Key: HDFS-10446
> URL: https://issues.apache.org/jira/browse/HDFS-10446
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> In a spirit of accuracy and correctness, async DFS APIs should also be tested 
> in the case of interleaving requests and responses. Especially, simulate 
> random order of async calls and random order of retrieving final results in 
> format of Future#get.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10445) Add timeout tests for async DFS API

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10445:
-
Summary: Add timeout tests for async DFS API  (was: Add timeout tests for 
asynchronous async DFS API)

> Add timeout tests for async DFS API
> ---
>
> Key: HDFS-10445
> URL: https://issues.apache.org/jira/browse/HDFS-10445
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> As a result of HADOOP-13168 commit, async DFS APIs should also be tested in 
> the case of timeout (i.e. Future#get(int timeout, TimeUnit unit)).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10445) Add timeout tests for asynchronous async DFS API

[Xiaobing Zhou (JIRA)]

Xiaobing Zhou created HDFS-10445:


 Summary: Add timeout tests for asynchronous async DFS API
 Key: HDFS-10445
 URL: https://issues.apache.org/jira/browse/HDFS-10445
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Xiaobing Zhou
Assignee: Xiaobing Zhou






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10445) Add timeout tests for asynchronous async DFS API

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10445:
-
Description: As a result of HADOOP-13168 commit, async DFS APIs should also 
be tested in the case of timeout (i.e. Future#get(int timeout, TimeUnit unit)). 
 (was: As a result of HADOOP-13168 commit, async DFS API should also be tested 
in the case of timeout (i.e. Future#get(int timeout, TimeUnit unit)).)

> Add timeout tests for asynchronous async DFS API
> 
>
> Key: HDFS-10445
> URL: https://issues.apache.org/jira/browse/HDFS-10445
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> As a result of HADOOP-13168 commit, async DFS APIs should also be tested in 
> the case of timeout (i.e. Future#get(int timeout, TimeUnit unit)).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10446) Add interleaving tests for asynchronous async DFS API

[Xiaobing Zhou (JIRA)]

Xiaobing Zhou created HDFS-10446:


 Summary: Add interleaving tests for asynchronous async DFS API
 Key: HDFS-10446
 URL: https://issues.apache.org/jira/browse/HDFS-10446
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Xiaobing Zhou
Assignee: Xiaobing Zhou


In a spirit of accuracy and correctness, async DFS APIs should also be tested 
in the case of interleaving requests and responses. Especially, simulate random 
order of async calls and random order of retrieving final results in format of 
Future#get.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10390) Implement asynchronous setAcl/getAclStatus for DistributedFileSystem

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294725#comment-15294725
 ] 

Xiaobing Zhou commented on HDFS-10390:
--

HDFS-10445 is filed to address timeout tests for all async DFS APIs to keep 
this patch neat.

> Implement asynchronous setAcl/getAclStatus for DistributedFileSystem
> 
>
> Key: HDFS-10390
> URL: https://issues.apache.org/jira/browse/HDFS-10390
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10390-HDFS-9924.000.patch, 
> HDFS-10390-HDFS-9924.001.patch, HDFS-10390-HDFS-9924.002.patch, 
> HDFS-10390-HDFS-9924.003.patch, HDFS-10390-HDFS-9924.004.patch, 
> HDFS-10390-HDFS-9924.005.patch, HDFS-10390-HDFS-9924.006.patch, 
> HDFS-10390-HDFS-9924.007.patch
>
>
> This is proposed to implement asynchronous setAcl/getAclStatus.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10445) Add timeout tests for asynchronous async DFS API

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10445:
-
Description: As a result of HADOOP-13168 commit, async DFS API should also 
be tested in the case of timeout (i.e. Future#get(int timeout, TimeUnit unit)).

> Add timeout tests for asynchronous async DFS API
> 
>
> Key: HDFS-10445
> URL: https://issues.apache.org/jira/browse/HDFS-10445
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> As a result of HADOOP-13168 commit, async DFS API should also be tested in 
> the case of timeout (i.e. Future#get(int timeout, TimeUnit unit)).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10408) Add out-of-order tests for async DFS API

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10408:
-
Summary: Add out-of-order tests for async DFS API  (was: Add tests for 
out-of-order asynchronous async DFS API)

> Add out-of-order tests for async DFS API
> 
>
> Key: HDFS-10408
> URL: https://issues.apache.org/jira/browse/HDFS-10408
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10408-HDFS-9924.000.patch
>
>
> HDFS-10224 and HDFS-10346 mostly test the batch style async request/response. 
> Out-of-order case (i.e. out of order retrieval of response) should also be 
> tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10408) Add tests for out-of-order asynchronous async DFS API

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10408:
-
Description: HDFS-10224 and HDFS-10346 mostly test the batch style async 
request/response. Out-of-order case (i.e. out of order retrieval of response) 
should also be tested.  (was: HDFS-10224 and HDFS-10346 mostly test the batch 
style async request/response. Out-of-order case should also be tested.)

> Add tests for out-of-order asynchronous async DFS API
> -
>
> Key: HDFS-10408
> URL: https://issues.apache.org/jira/browse/HDFS-10408
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10408-HDFS-9924.000.patch
>
>
> HDFS-10224 and HDFS-10346 mostly test the batch style async request/response. 
> Out-of-order case (i.e. out of order retrieval of response) should also be 
> tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10408) Add tests for out-of-order asynchronous async DFS API

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10408:
-
Summary: Add tests for out-of-order asynchronous async DFS API  (was: Add 
tests for out-of-order asynchronous rename/setPermission/setOwner)

> Add tests for out-of-order asynchronous async DFS API
> -
>
> Key: HDFS-10408
> URL: https://issues.apache.org/jira/browse/HDFS-10408
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10408-HDFS-9924.000.patch
>
>
> HDFS-10224 and HDFS-10346 mostly test the batch style async request/response. 
> Out-of-order case should also be tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10430) Refactor FileSystem#checkAccessPermissions for better reuse from tests

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294712#comment-15294712
 ] 

Xiaobing Zhou commented on HDFS-10430:
--

[~boky01] thank you for the update.

We can add/refactor something like this. It should not have security concerns. 
I will post a patch.
{code}
static void checkAccessPermissions(
  final FileStatus stat,
  final FsAction mode,
  final UserGroupInformation ugi) throws IOException {
{code}

> Refactor FileSystem#checkAccessPermissions for better reuse from tests
> --
>
> Key: HDFS-10430
> URL: https://issues.apache.org/jira/browse/HDFS-10430
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> FileSystem#checkAccessPermissions could be used in a bunch of tests from 
> different projects, but it's in hadoop-common, which is not visible in some 
> cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10390) Implement asynchronous setAcl/getAclStatus for DistributedFileSystem

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294706#comment-15294706
 ] 

Xiaobing Zhou commented on HDFS-10390:
--

[~eddyxu] thank you for comments. 
1. fail(""); is added in TestAsyncDFS#testAsyncAPIWithException to make 
sure the exception is indeed thrown.
2. HDFS-10444 is filed to do refactoring of TestAsyncDFSRename you suggested.

> Implement asynchronous setAcl/getAclStatus for DistributedFileSystem
> 
>
> Key: HDFS-10390
> URL: https://issues.apache.org/jira/browse/HDFS-10390
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10390-HDFS-9924.000.patch, 
> HDFS-10390-HDFS-9924.001.patch, HDFS-10390-HDFS-9924.002.patch, 
> HDFS-10390-HDFS-9924.003.patch, HDFS-10390-HDFS-9924.004.patch, 
> HDFS-10390-HDFS-9924.005.patch, HDFS-10390-HDFS-9924.006.patch, 
> HDFS-10390-HDFS-9924.007.patch
>
>
> This is proposed to implement asynchronous setAcl/getAclStatus.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10390) Implement asynchronous setAcl/getAclStatus for DistributedFileSystem

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294703#comment-15294703
 ] 

Xiaobing Zhou commented on HDFS-10390:
--

v007 patch is posted.
1. removed out-of-order test, since it will be addressed in HDFS-10408
2. fixed some issues as a result of HADOOP-13168 commit.
3. made minimum change to TestAsyncDFSRename
4. only restart MiniDFSCluster if necessary to accelerate tests.


> Implement asynchronous setAcl/getAclStatus for DistributedFileSystem
> 
>
> Key: HDFS-10390
> URL: https://issues.apache.org/jira/browse/HDFS-10390
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10390-HDFS-9924.000.patch, 
> HDFS-10390-HDFS-9924.001.patch, HDFS-10390-HDFS-9924.002.patch, 
> HDFS-10390-HDFS-9924.003.patch, HDFS-10390-HDFS-9924.004.patch, 
> HDFS-10390-HDFS-9924.005.patch, HDFS-10390-HDFS-9924.006.patch, 
> HDFS-10390-HDFS-9924.007.patch
>
>
> This is proposed to implement asynchronous setAcl/getAclStatus.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10390) Implement asynchronous setAcl/getAclStatus for DistributedFileSystem

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10390:
-
Attachment: HDFS-10390-HDFS-9924.007.patch

> Implement asynchronous setAcl/getAclStatus for DistributedFileSystem
> 
>
> Key: HDFS-10390
> URL: https://issues.apache.org/jira/browse/HDFS-10390
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10390-HDFS-9924.000.patch, 
> HDFS-10390-HDFS-9924.001.patch, HDFS-10390-HDFS-9924.002.patch, 
> HDFS-10390-HDFS-9924.003.patch, HDFS-10390-HDFS-9924.004.patch, 
> HDFS-10390-HDFS-9924.005.patch, HDFS-10390-HDFS-9924.006.patch, 
> HDFS-10390-HDFS-9924.007.patch
>
>
> This is proposed to implement asynchronous setAcl/getAclStatus.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10444) Refactor tests by moving irrevelant parts out of TestAsyncDFSRename

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294672#comment-15294672
 ] 

Xiaobing Zhou commented on HDFS-10444:
--

Per [this 
comment|https://issues.apache.org/jira/browse/HDFS-10390?focusedCommentId=15292269=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15292269]
 of HDFS-10390 from [~eddyxu], this jira tracks efforts to do necessary 
refactoring.

> Refactor tests by moving irrevelant parts out of TestAsyncDFSRename
> ---
>
> Key: HDFS-10444
> URL: https://issues.apache.org/jira/browse/HDFS-10444
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> TestAsyncDFSRename contains many tests related to setPermission, setOwner and 
> so on. They should be moved TestAsyncDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10444) Refactor tests by moving irrevelant parts out of TestAsyncDFSRename

[Xiaobing Zhou (JIRA)]

Xiaobing Zhou created HDFS-10444:


 Summary: Refactor tests by moving irrevelant parts out of 
TestAsyncDFSRename
 Key: HDFS-10444
 URL: https://issues.apache.org/jira/browse/HDFS-10444
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Xiaobing Zhou
Assignee: Xiaobing Zhou


TestAsyncDFSRename contains many tests related to setPermission, setOwner and 
so on. They should be moved TestAsyncDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8057) Move BlockReader implementation to the client implementation package

[Tsz Wo Nicholas Sze (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294657#comment-15294657
 ] 

Tsz Wo Nicholas Sze commented on HDFS-8057:
---

Thanks for upload the branch-2 patch.  There are some failed tests.  Could you 
take a look?

> Move BlockReader implementation to the client implementation package
> 
>
> Key: HDFS-8057
> URL: https://issues.apache.org/jira/browse/HDFS-8057
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Takanobu Asanuma
> Attachments: HDFS-8057.1.patch, HDFS-8057.2.patch, HDFS-8057.3.patch, 
> HDFS-8057.4.patch, HDFS-8057.branch-2.001.patch, HDFS-8057.branch-2.5.patch
>
>
> BlockReaderLocal, RemoteBlockReader, etc should be moved to 
> org.apache.hadoop.hdfs.client.impl.  We may as well rename RemoteBlockReader 
> to BlockReaderRemote.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10403) DiskBalancer: Add cancel command

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294589#comment-15294589
 ] 

Hadoop QA commented on HDFS-10403:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
13s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} HDFS-1312 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s 
{color} | {color:green} HDFS-1312 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
33s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s 
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s 
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s 
{color} | {color:green} HDFS-1312 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s 
{color} | {color:green} HDFS-1312 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s 
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s 
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 48s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 54m 17s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
26s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 143m 6s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_91 Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| JDK v1.7.0_101 Failed junit tests | hadoop.tools.TestHdfsConfigFields |
|   | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:cf2ee45 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12805317/HDFS-10403-HDFS-1312.002.patch
 |
| JIRA Issue | HDFS-10403 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  

[jira] [Commented] (HDFS-10378) FSDirAttrOp#setOwner throws AccessControlException with wrong message

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294408#comment-15294408
 ] 

Hadoop QA commented on HDFS-10378:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
42s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
24s {color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: patch generated 0 
new + 24 unchanged - 3 fixed = 24 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 43s {color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
22s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 8s {color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
|   | hadoop.hdfs.TestAsyncDFSRename |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12805303/HDFS-10378.002.patch |
| JIRA Issue | HDFS-10378 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 15c007f3e19b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 500e946 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/15515/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HDFS-Build/15515/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/15515/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/15515/console |
| Powered by | Apache Yetus 0.2.0   http://yetus.apache.org |


This message was automatically generated.



> FSDirAttrOp#setOwner throws AccessControlException with wrong message
> -

[jira] [Updated] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-10420:
-
  Resolution: Fixed
Hadoop Flags: Reviewed
   Fix Version/s: HDFS-7240
Target Version/s:   (was: HDFS-7240)
  Status: Resolved  (was: Patch Available)

Committed to the feature branch. Thank you for the reviews [~anu]!

> Fix Ozone unit tests to use MiniOzoneCluster
> 
>
> Key: HDFS-10420
> URL: https://issues.apache.org/jira/browse/HDFS-10420
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone, test
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: HDFS-7240
>
> Attachments: HDFS-10420-HDFS-7240.01.patch, 
> HDFS-10420-HDFS-7240.02.patch, HDFS-10420-HDFS-7240.03.patch, 
> HDFS-10420-HDFS-7240.04.patch, HDFS-10420-HDFS-7240.05.patch
>
>
> [~cnauroth] added MiniOzoneCluster support to the Ozone feature branch. We 
> should update all the Ozone tests to use it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster

[Arpit Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294377#comment-15294377
 ] 

Arpit Agarwal commented on HDFS-10420:
--

The javac warning is not in code introduced by this patch and the 
TestOzoneRestWithMiniCluster failure looks spurious. I was unable to reproduce 
it on either Mac OS X or on Centos 6.

I will commit this patch shortly.

> Fix Ozone unit tests to use MiniOzoneCluster
> 
>
> Key: HDFS-10420
> URL: https://issues.apache.org/jira/browse/HDFS-10420
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone, test
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-10420-HDFS-7240.01.patch, 
> HDFS-10420-HDFS-7240.02.patch, HDFS-10420-HDFS-7240.03.patch, 
> HDFS-10420-HDFS-7240.04.patch, HDFS-10420-HDFS-7240.05.patch
>
>
> [~cnauroth] added MiniOzoneCluster support to the Ozone feature branch. We 
> should update all the Ozone tests to use it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10403) DiskBalancer: Add cancel command

[Arpit Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294356#comment-15294356
 ] 

Arpit Agarwal commented on HDFS-10403:
--

+1 thanks for the update Anu.

A minor thing that can be addressed at commit time - I think 
CancelCommand#getHelp needs to be updated.

> DiskBalancer: Add cancel  command
> -
>
> Key: HDFS-10403
> URL: https://issues.apache.org/jira/browse/HDFS-10403
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-1312
>
> Attachments: HDFS-10403-HDFS-1312.001.patch, 
> HDFS-10403-HDFS-1312.002.patch
>
>
> Allows user to cancel an on-going disk balancing operation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10403) DiskBalancer: Add cancel command

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-10403:
-
Status: Patch Available  (was: Open)

> DiskBalancer: Add cancel  command
> -
>
> Key: HDFS-10403
> URL: https://issues.apache.org/jira/browse/HDFS-10403
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-1312
>
> Attachments: HDFS-10403-HDFS-1312.001.patch, 
> HDFS-10403-HDFS-1312.002.patch
>
>
> Allows user to cancel an on-going disk balancing operation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support

[Bob Hansen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294337#comment-15294337
 ] 

Bob Hansen commented on HDFS-10441:
---

Notes in chronological order; most of these are minor, nitpicky points.

* Should Options::ha_namenodes_ be a map to make it 
easier to look up?

* Rather than have a server_exception_type, I would like to see the explicit 
error types as peers to our other error types.  The fact that it came from a 
Java exception on the server side is an implementation detail of the 
communications.  The kExceptionType should really be an "UnexpectedException" 
type, and all of the others should be mapped to semantically informative types 
(using an awesome lookup table, like you did)

* Not super-critical, but it's a little bit cleaner and (possibly) faster to 
use strstr in HDFSConfiguration::SplitOnComma.  Once you know the start and 
end, you can use std::string::string(const char *, int) to construct the string.

* Instead of trying to manually parse the URI in ParsePartialURI, would it be 
easier to do a quick check for "://", and if it's not there, prepend 
DEFAULT_SCHEME (hdfs://), then pass it to the full-featured URI parser?

* By design the Configuration class is really a stupid container.  If we can, 
all the logic should be in the HDFSConfigurationLoader.

* It seems that the nameservice lookups aren't actually URIs; they're just 
server:port addresses.  Perphaps rather than storing a URI object, the 
HANamenodeInfo should just have a server and port field.  The rest of the URI 
semantic space (scheme, user, password, path, fragment, etc.) really doesn't 
apply here, and would be an error if it appeared in the config file.

* I would prefer "FormatURI" or "ToString" over "FmtURI" as a method name

* I've noticed that we've grown used to "if (!stat.ok) { LOG_XXX(..., )".  In trying to debug problems, I'm unhappy with me for not 
including the error message.  As you touch any logging messages, << out the 
status.ToString() also

* As an extension of that: LOG_WARN(kRPC, << "Tried to connect to standby!");  
What node was I trying to connect to?  

* EnqueueRequestsWithLock: I think elsewhere in the codebase we have used 
EnqueueRequests_locked; let's keep that standard

* is_locked: we already have a lock_held method in util.h (that works with 
reentrant mutexes, too)

* I know we're going to add more-than-one HA NN later, but a good idiom is to 
keep them in a sorted vector, and have an index pointing to the current one.  
Then make next = (curr + 1) % size.

* Name lookup in ResoloveAllEndpoints needs to be async.  This can be follow-up 
if necessary.

* In general, having a recursive_mutex (HA NN Tracker) is a bad smell that 
we're not thinking about concurrency tightly enough.  I'm not sure that's the 
case here, but let's think about it.

* Make sure to clean up TODOs and anything referring to internal server names

* Unset port should be a WARNING, if we're going to default and move on

* In RpcEngine::Start, if resolution fails and we clear out ha_persisted_info_, 
what will happen when we run?

* Better logging than "" and "Caught this early"

* In RpcEngine::RpcCommsError, we should try for symmetry in retry count and 
failure count.  Let's pull the IncrementRetryCount out of being inline, and 
just use both getters in the call to ShouldRetry.

* You have a comment for eventually failing out messages if we're bouncing 
between standby nodes, but it looks like we only actually check if it's not a 
kStandbyException.  If I'm mis-reading, let me know.

* We don't need the first line here, do we?
+  last_endpoints_.clear();
+  last_endpoints_ = new_active_nn_info.endpoints;

* In RpcConnection::HandleRpcResponse, I think we should be checking status and 
erroring out before calling RemoveFromRunningQueue.  We should look at this 
closely, and then we could get rid of the EnqueueReqeustsWithLock method.

> libhdfs++: HA namenode support
> --
>
> Key: HDFS-10441
> URL: https://issues.apache.org/jira/browse/HDFS-10441
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
> Attachments: HDFS-10441.HDFS-8707.000.patch
>
>
> If a cluster is HA enabled then do proper failover.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10443) FairCallQueue makes callQueue larger than the configured capacity.

[Konstantin Shvachko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294326#comment-15294326
 ] 

Konstantin Shvachko commented on HDFS-10443:


FairCallQueue was introduced by HADOOP-10282.

> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HDFS-10443
> URL: https://issues.apache.org/jira/browse/HDFS-10443
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default) 
> sub-queues, with each sub-queue corresponding to a different level of 
> priority. The constructor for {{FairCallQueue}} takes the same parameter 
> {{capacity}} as the default CallQueue implementation, and allocates all its 
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by 
> default it results in the total callQueue size 4 times larger than it should 
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294321#comment-15294321
 ] 

Hadoop QA commented on HDFS-10441:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color} 
| {color:red} HDFS-10441 does not apply to HDFS-8707. Rebase required? Wrong 
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12805246/HDFS-10441.HDFS-8707.000.patch
 |
| JIRA Issue | HDFS-10441 |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/15516/console |
| Powered by | Apache Yetus 0.2.0   http://yetus.apache.org |


This message was automatically generated.



> libhdfs++: HA namenode support
> --
>
> Key: HDFS-10441
> URL: https://issues.apache.org/jira/browse/HDFS-10441
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
> Attachments: HDFS-10441.HDFS-8707.000.patch
>
>
> If a cluster is HA enabled then do proper failover.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10443) FairCallQueue makes callQueue larger than the configured capacity.

[Konstantin Shvachko (JIRA)]

Konstantin Shvachko created HDFS-10443:
--

 Summary: FairCallQueue makes callQueue larger than the configured 
capacity.
 Key: HDFS-10443
 URL: https://issues.apache.org/jira/browse/HDFS-10443
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.6.0
Reporter: Konstantin Shvachko


{{FairCallQueue}} divides {{callQueue}} into multiple (4 by default) 
sub-queues, with each sub-queue corresponding to a different level of priority. 
The constructor for {{FairCallQueue}} takes the same parameter {{capacity}} as 
the default CallQueue implementation, and allocates all its sub-queues of size 
{{capacity}}. With 4 levels of priority (sub-queues) by default it results in 
the total callQueue size 4 times larger than it should be based on the 
configuration.
{{capacity}} should be divided by the number of sub-queues at some place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10403) DiskBalancer: Add cancel command

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-10403:

Attachment: HDFS-10403-HDFS-1312.002.patch

> DiskBalancer: Add cancel  command
> -
>
> Key: HDFS-10403
> URL: https://issues.apache.org/jira/browse/HDFS-10403
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-1312
>
> Attachments: HDFS-10403-HDFS-1312.001.patch, 
> HDFS-10403-HDFS-1312.002.patch
>
>
> Allows user to cancel an on-going disk balancing operation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10441) libhdfs++: HA namenode support

[James Clampffer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Clampffer updated HDFS-10441:
---
Status: Patch Available  (was: Open)

> libhdfs++: HA namenode support
> --
>
> Key: HDFS-10441
> URL: https://issues.apache.org/jira/browse/HDFS-10441
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
> Attachments: HDFS-10441.HDFS-8707.000.patch
>
>
> If a cluster is HA enabled then do proper failover.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10188) libhdfs++: Implement debug allocators

[James Clampffer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Clampffer updated HDFS-10188:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> libhdfs++: Implement debug allocators
> -
>
> Key: HDFS-10188
> URL: https://issues.apache.org/jira/browse/HDFS-10188
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: Xiaowei Zhu
> Attachments: HDFS-10188.HDFS-8707.000.patch, 
> HDFS-10188.HDFS-8707.001.patch, HDFS-10188.HDFS-8707.002.patch, 
> HDFS-10188.HDFS-8707.003.patch, HDFS-10188.HDFS-8707.004.patch
>
>
> I propose implementing a set of memory new/delete pairs with additional 
> checking to detect double deletes, read-after-delete, and write-after-deletes 
> to help debug resource ownership issues and prevent new ones from entering 
> the library.
> One of the most common issues we have is use-after-free issues.  The 
> continuation pattern makes these really tricky to debug because by the time a 
> segsegv is raised the context of what has caused the error is long gone.
> The plan is to add allocators that can be turned on that can do the 
> following, in order of runtime cost.
> 1: no-op, forward through to default new/delete
> 2: make sure the memory given to the constructor is dirty, memset free'd 
> memory to 0
> 3: implement operator new with mmap, lock that region of memory once it's 
> been deleted; obviously this can't be left to run forever because the memory 
> is never unmapped
> This should also put some groundwork in place for implementing specialized 
> allocators for tiny objects that we churn through like std::string.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10188) libhdfs++: Implement debug allocators

[James Clampffer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294301#comment-15294301
 ] 

James Clampffer commented on HDFS-10188:


Committed to HDFS-8707.  Thanks for the patch Xiaowei

> libhdfs++: Implement debug allocators
> -
>
> Key: HDFS-10188
> URL: https://issues.apache.org/jira/browse/HDFS-10188
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: Xiaowei Zhu
> Attachments: HDFS-10188.HDFS-8707.000.patch, 
> HDFS-10188.HDFS-8707.001.patch, HDFS-10188.HDFS-8707.002.patch, 
> HDFS-10188.HDFS-8707.003.patch, HDFS-10188.HDFS-8707.004.patch
>
>
> I propose implementing a set of memory new/delete pairs with additional 
> checking to detect double deletes, read-after-delete, and write-after-deletes 
> to help debug resource ownership issues and prevent new ones from entering 
> the library.
> One of the most common issues we have is use-after-free issues.  The 
> continuation pattern makes these really tricky to debug because by the time a 
> segsegv is raised the context of what has caused the error is long gone.
> The plan is to add allocators that can be turned on that can do the 
> following, in order of runtime cost.
> 1: no-op, forward through to default new/delete
> 2: make sure the memory given to the constructor is dirty, memset free'd 
> memory to 0
> 3: implement operator new with mmap, lock that region of memory once it's 
> been deleted; obviously this can't be left to run forever because the memory 
> is never unmapped
> This should also put some groundwork in place for implementing specialized 
> allocators for tiny objects that we churn through like std::string.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10402) DiskBalancer: Add QueryStatus command

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-10402:
-
  Resolution: Fixed
Target Version/s:   (was: HDFS-1312)
  Status: Resolved  (was: Patch Available)

I committed this to the feature branch. Thank you for the contribution [~anu].

> DiskBalancer: Add QueryStatus command
> -
>
> Key: HDFS-10402
> URL: https://issues.apache.org/jira/browse/HDFS-10402
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-1312
>
> Attachments: HDFS-10402-HDFS-1312.001.patch, 
> HDFS-10402-HDFS-1312.002.patch
>
>
> Adds QueryStatus which gets disk balancer status from a specific node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10403) DiskBalancer: Add cancel command

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-10403:

Attachment: (was: HDFS-10403-HDFS-1312.002.patch)

> DiskBalancer: Add cancel  command
> -
>
> Key: HDFS-10403
> URL: https://issues.apache.org/jira/browse/HDFS-10403
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-1312
>
> Attachments: HDFS-10403-HDFS-1312.001.patch
>
>
> Allows user to cancel an on-going disk balancing operation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10378) FSDirAttrOp#setOwner throws AccessControlException with wrong message

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HDFS-10378:
--
Attachment: HDFS-10378.002.patch

Patch 002:
* Fix {{setOwner}} permission checking code
* Enhance unit tests
* Use JUnit4 {{ExpectedException}} to simplify exception expecting unit tests

> FSDirAttrOp#setOwner throws AccessControlException with wrong message
> -
>
> Key: HDFS-10378
> URL: https://issues.apache.org/jira/browse/HDFS-10378
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>  Labels: supportability
> Attachments: HDFS-10378-unit.patch, HDFS-10378.001.patch, 
> HDFS-10378.002.patch
>
>
> Calling {{setOwner}} as a non-super user does trigger 
> {{AccessControlException}}, however, the message "Permission denied. 
> user=user1967821757 is not the owner of inode=child" is wrong. Expect this 
> message: "Non-super user cannot change owner".
> Output of patched unit test {{TestPermission.testFilePermission}}:
> {noformat}
> 2016-05-06 16:45:44,915 [main] INFO  security.TestPermission 
> (TestPermission.java:testFilePermission(280)) - GOOD: got 
> org.apache.hadoop.security.AccessControlException: Permission denied. 
> user=user1967821757 is not the owner of inode=child1
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:273)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:250)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1642)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1626)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:1595)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1717)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:835)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:481)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:665)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2423)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2419)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1755)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2417)
> {noformat}
> Will upload the unit test patch shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10438) When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294113#comment-15294113
 ] 

Hudson commented on HDFS-10438:
---

SUCCESS: Integrated in Hadoop-trunk-Commit #9835 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/9835/])
HDFS-10438. When NameNode HA is configured to use the lifeline RPC (cnauroth: 
rev 500e946729e1b85fdc0ce8475d8ce118ba886f46)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestNNHealthCheck.java


> When NameNode HA is configured to use the lifeline RPC server, it should log 
> the address of that server.
> 
>
> Key: HDFS-10438
> URL: https://issues.apache.org/jira/browse/HDFS-10438
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: ha, namenode
>Reporter: KWON BYUNGCHANG
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-10438.001.patch, HDFS-10438.002.patch
>
>
> As reported by [~magnum]:
> I have configured below
> {code}
> dfs.namenode.servicerpc-address.xdev.nn1=my.host.com:8040
> dfs.namenode.lifeline.rpc-address.xdev.nn1=my.host.com:8041
> {code}
> servicerpc port is 8040,  lifeline port is 8041.
> however zkfc daemon is logging using servicerpc port. 
> It may cause confusion.
> thank you.
> {code}
> 2016-05-19 19:18:40,566 WARN  ha.HealthMonitor 
> (HealthMonitor.java:doHealthChecks(207)) - Service health check failed for 
> NameNode at my.host.com/10.114.87.91:8040: The NameNode has no resources 
> available
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10424) DatanodeLifelineProtocol not able to use under security cluster

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294112#comment-15294112
 ] 

Hudson commented on HDFS-10424:
---

SUCCESS: Integrated in Hadoop-trunk-Commit #9835 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/9835/])
HDFS-10424. DatanodeLifelineProtocol not able to use under security (cnauroth: 
rev bcde1562d25c4f5595f4e3436dc3630315b1ceed)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HDFSPolicyProvider.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHDFSPolicyProvider.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNodeRpcServer.java


> DatanodeLifelineProtocol not able to use under security cluster
> ---
>
> Key: HDFS-10424
> URL: https://issues.apache.org/jira/browse/HDFS-10424
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: gu-chi
>Assignee: Chris Nauroth
>Priority: Blocker
> Fix For: 2.8.0
>
> Attachments: HDFS-10424-branch-2.8.001.patch, HDFS-10424.001.patch
>
>
> {quote}
> protocol org.apache.hadoop.hdfs.server.protocol.DatanodeLifelineProtocol is 
> unauthorized for user * (auth:KERBEROS) | Server.java:1979
> {quote}
> I am using security cluster authenticate with kerberos, as I checked the the 
> code, if security auth enabled, because the DatanodeLifelineProtocol is not 
> inside HDFSPolicyProvider, when authorize in ServiceAuthorizationManager, 
> AuthorizationException will be thrown at line 96.
> Please point me out if I am wrong



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10402) DiskBalancer: Add QueryStatus command

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294110#comment-15294110
 ] 

Hadoop QA commented on HDFS-10402:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m 27s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
23s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s 
{color} | {color:green} HDFS-1312 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s 
{color} | {color:green} HDFS-1312 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
27s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s 
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s 
{color} | {color:green} HDFS-1312 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s 
{color} | {color:green} HDFS-1312 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s 
{color} | {color:green} HDFS-1312 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s 
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s 
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 19s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 20s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
24s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 189m 51s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_91 Failed junit tests | hadoop.tools.TestHdfsConfigFields |
|   | hadoop.hdfs.TestAsyncDFSRename |
| JDK v1.7.0_101 Failed junit tests | hadoop.tools.TestHdfsConfigFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:cf2ee45 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12805242/HDFS-10402-HDFS-1312.002.patch
 |
| JIRA Issue | HDFS-10402 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  

[jira] [Updated] (HDFS-10438) When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.

[Chris Nauroth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HDFS-10438:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
   Status: Resolved  (was: Patch Available)

I have committed this to trunk, branch-2 and branch-2.8.  [~arpitagarwal] and 
[~kihwal], thank you for the code reviews.

> When NameNode HA is configured to use the lifeline RPC server, it should log 
> the address of that server.
> 
>
> Key: HDFS-10438
> URL: https://issues.apache.org/jira/browse/HDFS-10438
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: ha, namenode
>Reporter: KWON BYUNGCHANG
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-10438.001.patch, HDFS-10438.002.patch
>
>
> As reported by [~magnum]:
> I have configured below
> {code}
> dfs.namenode.servicerpc-address.xdev.nn1=my.host.com:8040
> dfs.namenode.lifeline.rpc-address.xdev.nn1=my.host.com:8041
> {code}
> servicerpc port is 8040,  lifeline port is 8041.
> however zkfc daemon is logging using servicerpc port. 
> It may cause confusion.
> thank you.
> {code}
> 2016-05-19 19:18:40,566 WARN  ha.HealthMonitor 
> (HealthMonitor.java:doHealthChecks(207)) - Service health check failed for 
> NameNode at my.host.com/10.114.87.91:8040: The NameNode has no resources 
> available
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10424) DatanodeLifelineProtocol not able to use under security cluster

[Chris Nauroth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HDFS-10424:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
   Status: Resolved  (was: Patch Available)

[~arpitagarwal] and [~vinayrpet], thank you for the code reviews.  I have 
committed this to trunk, branch-2 and branch-2.8.

> DatanodeLifelineProtocol not able to use under security cluster
> ---
>
> Key: HDFS-10424
> URL: https://issues.apache.org/jira/browse/HDFS-10424
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: gu-chi
>Assignee: Chris Nauroth
>Priority: Blocker
> Fix For: 2.8.0
>
> Attachments: HDFS-10424-branch-2.8.001.patch, HDFS-10424.001.patch
>
>
> {quote}
> protocol org.apache.hadoop.hdfs.server.protocol.DatanodeLifelineProtocol is 
> unauthorized for user * (auth:KERBEROS) | Server.java:1979
> {quote}
> I am using security cluster authenticate with kerberos, as I checked the the 
> code, if security auth enabled, because the DatanodeLifelineProtocol is not 
> inside HDFSPolicyProvider, when authorize in ServiceAuthorizationManager, 
> AuthorizationException will be thrown at line 96.
> Please point me out if I am wrong



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9924) [umbrella] Asynchronous HDFS Access

[Colin Patrick McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294073#comment-15294073
 ] 

Colin Patrick McCabe commented on HDFS-9924:


bq. Good point! On the other hand, programmers who are NOT familiar with 
Node.js may NOT want something that supports callback chaining.

Callback chaining is an optional feature which nobody is forced to use.  I 
don't see why anyone would prefer Future over CompletableFuture.

bq. Also, you might not have noticed, supporting Future is a step toward 
supporting CompletableFuture.

I don't see why supporting Future is a step towards supporting a different API. 
 I think Hadoop has too many APIs with duplicate functionality already, and we 
should try to minimize the cognitive load on new developers.

bq. I guess you might have misunderstood the release process. The release 
manager could include/exclude any feature as she/he pleases.

Which 2.x release do you want this to become part of?

> [umbrella] Asynchronous HDFS Access
> ---
>
> Key: HDFS-9924
> URL: https://issues.apache.org/jira/browse/HDFS-9924
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: AsyncHdfs20160510.pdf
>
>
> This is an umbrella JIRA for supporting Asynchronous HDFS Access.
> Currently, all the API methods are blocking calls -- the caller is blocked 
> until the method returns.  It is very slow if a client makes a large number 
> of independent calls in a single thread since each call has to wait until the 
> previous call is finished.  It is inefficient if a client needs to create a 
> large number of threads to invoke the calls.
> We propose adding a new API to support asynchronous calls, i.e. the caller is 
> not blocked.  The methods in the new API immediately return a Java Future 
> object.  The return value can be obtained by the usual Future.get() method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-7240) Object store in HDFS

[Arpit Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294022#comment-15294022
 ] 

Arpit Agarwal commented on HDFS-7240:
-

Andrew, this tone is not helpful. Nothing we presented at ApacheCon (also an 
Apache forum) was a significant change from the architecture doc. We will post 
a design doc soon and there is ample opportunity/need for community 
contributions as the implementation is still in an early stage. This is in line 
with how features are developed in Apache.

> Object store in HDFS
> 
>
> Key: HDFS-7240
> URL: https://issues.apache.org/jira/browse/HDFS-7240
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: Ozone-architecture-v1.pdf, ozone_user_v0.pdf
>
>
> This jira proposes to add object store capabilities into HDFS. 
> As part of the federation work (HDFS-1052) we separated block storage as a 
> generic storage layer. Using the Block Pool abstraction, new kinds of 
> namespaces can be built on top of the storage layer i.e. datanodes.
> In this jira I will explore building an object store using the datanode 
> storage, but independent of namespace metadata.
> I will soon update with a detailed design document.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10403) DiskBalancer: Add cancel command

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-10403:

Attachment: HDFS-10403-HDFS-1312.002.patch

[~arpitagarwal] Thanks for your review comments. This patch supports cancelling 
a plan using node address and plan ID.

> DiskBalancer: Add cancel  command
> -
>
> Key: HDFS-10403
> URL: https://issues.apache.org/jira/browse/HDFS-10403
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-1312
>
> Attachments: HDFS-10403-HDFS-1312.001.patch, 
> HDFS-10403-HDFS-1312.002.patch
>
>
> Allows user to cancel an on-going disk balancing operation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9924) [umbrella] Asynchronous HDFS Access

[Tsz Wo Nicholas Sze (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293942#comment-15293942
 ] 

Tsz Wo Nicholas Sze commented on HDFS-9924:
---

> ... The Hadoop 2.8 release has been blocked for a very, very long time. ...

I guess you might have misunderstood the release process.  The release manager 
could include/exclude any feature as she/he pleases.

> Programmers who are familiar with Node.js will want something that supports 
> callback chaining, 

Good point!  On the other hand, programmers who are NOT familiar with Node.js 
may NOT want something that supports callback chaining.

Also, you might not have noticed, supporting Future is a step toward supporting 
CompletableFuture.

> If we are going to backport this to branch-2, we should do it once the 
> feature is done, rather than backporting bits and pieces as we go. ...

I disagree.  Async HDFS is a collections of async methods.  We don't not have 
to support all the methods at day one.

> [umbrella] Asynchronous HDFS Access
> ---
>
> Key: HDFS-9924
> URL: https://issues.apache.org/jira/browse/HDFS-9924
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: AsyncHdfs20160510.pdf
>
>
> This is an umbrella JIRA for supporting Asynchronous HDFS Access.
> Currently, all the API methods are blocking calls -- the caller is blocked 
> until the method returns.  It is very slow if a client makes a large number 
> of independent calls in a single thread since each call has to wait until the 
> previous call is finished.  It is inefficient if a client needs to create a 
> large number of threads to invoke the calls.
> We propose adding a new API to support asynchronous calls, i.e. the caller is 
> not blocked.  The methods in the new API immediately return a Java Future 
> object.  The return value can be obtained by the usual Future.get() method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10402) DiskBalancer: Add QueryStatus command

[Arpit Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293890#comment-15293890
 ] 

Arpit Agarwal commented on HDFS-10402:
--

+1 pending Jenkins. Thanks [~anu].

> DiskBalancer: Add QueryStatus command
> -
>
> Key: HDFS-10402
> URL: https://issues.apache.org/jira/browse/HDFS-10402
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-1312
>
> Attachments: HDFS-10402-HDFS-1312.001.patch, 
> HDFS-10402-HDFS-1312.002.patch
>
>
> Adds QueryStatus which gets disk balancer status from a specific node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10385) LocalFileSystem rename() function should return false when destination file exists

[Chris Nauroth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293881#comment-15293881
 ] 

Chris Nauroth commented on HDFS-10385:
--

[~boky01], I agree.  I'm generally reluctant to change behavior of the local 
file system classes in the 2.x line.  If it's helpful, we could consider 
possible changes for trunk/3.x and mark them as backwards-incompatible.

> LocalFileSystem rename() function should return false when destination file 
> exists
> --
>
> Key: HDFS-10385
> URL: https://issues.apache.org/jira/browse/HDFS-10385
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.6.0
>Reporter: Aihua Xu
>Assignee: Xiaobing Zhou
>
> Currently rename() of LocalFileSystem returns true and renames successfully 
> when the destination file exists. That seems to have different behavior from 
> DFSFileSystem. 
> If they can have the same behavior, then we can use one call to do rename 
> rather than checking if destination exists and then making rename() call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10442) S3A file-create should throw error rather than overwrite directories

[Raymie Stata (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293845#comment-15293845
 ] 

Raymie Stata commented on HDFS-10442:
-

[~ste...@apache.org] found the following test-cases being skipped that probably 
shouldn't be:
{noformat}
2016-05-20 11:03:07,144 INFO  contract.AbstractFSContractTestBase 
(AbstractFSContractTestBase.java:describe(240)) - verify trying to create a 
file over a non-empty dir fails
2016-05-20 11:03:10,615 INFO  contract.ContractTestUtils 
(ContractTestUtils.java:skip(425)) - Skipping: Object store allows a file to 
overwrite a directory

2016-05-20 11:03:14,992 INFO  contract.AbstractFSContractTestBase 
(AbstractFSContractTestBase.java:setup(172)) - Test filesystem = 
s3a://hwdev-steve implemented by S3AFileSystem{uri=s3a://hwdev-steve, 
workingDir=s3a://hwdev-steve/user/stevel, partSize=104857600, 
enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, 
blockSize=33554432, multiPartThreshold=2147483647, statistics {0 bytes read, 
266 bytes written, 25 read ops, 0 large read ops, 11 write ops}, metrics 
{{Context=S3AFileSystem} 
{FileSystemId=0b9a969b-9c2e-48da-9bce-4599ae8a2cb9-hwdev-steve} 
{fsURI=s3a://hwdev-steve/} {files_created=3} {files_copied=0} 
{files_copied_bytes=0} {files_deleted=3} {directories_created=3} 
{directories_deleted=0} {ignored_errors=3} {streamReadOperations=0} 
{streamForwardSeekOperations=0} {streamBytesRead=0} {streamSeekOperations=0} 
{streamReadExceptions=0} {streamOpened=0} {streamReadOperationsIncomplete=0} 
{streamAborted=0} {streamReadFullyOperations=0} {streamClosed=0} 
{streamBytesSkippedOnSeek=0} {streamCloseOperations=0} 
{streamBytesBackwardsOnSeek=0} {streamBackwardSeekOperations=0} }}
2016-05-20 11:03:16,381 INFO  contract.ContractTestUtils 
(ContractTestUtils.java:skip(425)) - Skipping: blobstores can't distinguish 
empty directories from files
{noformat}

> S3A file-create should throw error rather than overwrite directories
> 
>
> Key: HDFS-10442
> URL: https://issues.apache.org/jira/browse/HDFS-10442
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Raymie Stata
>
> S3A.create(Path,FsPermission,boolean,int,short,long,Progressable) is not 
> checking to see if it's being asked to overwrite a directory.  It could 
> easily do so, and should throw an error in this case.
> There is a test-case for this in AbstractFSContractTestBase, but it's being 
> skipped because S3A is a blobstore.  However, both the Azure and Swift file 
> systems make this test, and the new S3 one should as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10442) S3A file-create should throw error rather than overwrite directories

[Raymie Stata (JIRA)]

Raymie Stata created HDFS-10442:
---

 Summary: S3A file-create should throw error rather than overwrite 
directories
 Key: HDFS-10442
 URL: https://issues.apache.org/jira/browse/HDFS-10442
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Raymie Stata


S3A.create(Path,FsPermission,boolean,int,short,long,Progressable) is not 
checking to see if it's being asked to overwrite a directory.  It could easily 
do so, and should throw an error in this case.

There is a test-case for this in AbstractFSContractTestBase, but it's being 
skipped because S3A is a blobstore.  However, both the Azure and Swift file 
systems make this test, and the new S3 one should as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-7240) Object store in HDFS

[Colin Patrick McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293822#comment-15293822
 ] 

Colin Patrick McCabe commented on HDFS-7240:


bq. Another question about reading the ApacheCon slides: the question "Why an 
Object Store" was well answered. How about "why an object store as part of 
HDFS"? IIUC Ozone is only leveraging a very small portion of HDFS code. Why 
should it be a part of HDFS instead of a separate project?

That's a very good question.  Why can't ozone be its own subproject within 
Hadoop?  We could add a hadoop-ozone directory at the top level of the git 
repo.  Ozone seems to be reusing very little of the HDFS code.  For example, it 
doesn't store blocks the way the DataNode stores blocks.  It doesn't run the 
HDFS NameNode.  It doesn't use the HDFS client code.

> Object store in HDFS
> 
>
> Key: HDFS-7240
> URL: https://issues.apache.org/jira/browse/HDFS-7240
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: Ozone-architecture-v1.pdf, ozone_user_v0.pdf
>
>
> This jira proposes to add object store capabilities into HDFS. 
> As part of the federation work (HDFS-1052) we separated block storage as a 
> generic storage layer. Using the Block Pool abstraction, new kinds of 
> namespaces can be built on top of the storage layer i.e. datanodes.
> In this jira I will explore building an object store using the datanode 
> storage, but independent of namespace metadata.
> I will soon update with a detailed design document.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9924) [umbrella] Asynchronous HDFS Access

[Colin Patrick McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293809#comment-15293809
 ] 

Colin Patrick McCabe commented on HDFS-9924:


I have to agree with [~andrew.wang] that it makes more sense to put these 
changes in trunk than in branch-2.  The Hadoop 2.8 release has been blocked for 
a very, very long time.  There are tons of features in branch-2 that have been 
waiting for almost a year to be released.  Adding yet another feature to 
branch-2, when we're so far behind on releases, doesn't make sense.

Programmers who are familiar with Node.js will want something that supports 
callback chaining, like CompletableFuture, rather than something like the 
old-style Future API.  If we target this at branch-3, we can use the jdk8 
CompletableFuture.

If we are going to backport this to branch-2, we should do it once the feature 
is done, rather than backporting bits and pieces as we go.  This is especially 
true when there are still open questions about the API.

> [umbrella] Asynchronous HDFS Access
> ---
>
> Key: HDFS-9924
> URL: https://issues.apache.org/jira/browse/HDFS-9924
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: AsyncHdfs20160510.pdf
>
>
> This is an umbrella JIRA for supporting Asynchronous HDFS Access.
> Currently, all the API methods are blocking calls -- the caller is blocked 
> until the method returns.  It is very slow if a client makes a large number 
> of independent calls in a single thread since each call has to wait until the 
> previous call is finished.  It is inefficient if a client needs to create a 
> large number of threads to invoke the calls.
> We propose adding a new API to support asynchronous calls, i.e. the caller is 
> not blocked.  The methods in the new API immediately return a Java Future 
> object.  The return value can be obtained by the usual Future.get() method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10441) libhdfs++: HA namenode support

[James Clampffer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Clampffer updated HDFS-10441:
---
Attachment: HDFS-10441.HDFS-8707.000.patch

Attached basic implementation patch.  Seems to work but needs lots of stuff 
cleaned up and improved.  Can't assign bugs but I'll claim this one.

-Moved DNS lookup to RPCEngine, but FileSystem will still do a round of it 
first.  Needs to be cleaned up
-Only 1 nameservice supported in config files.  If more than one exists the 
leftmost one in the file gets picked.
-RetryWithFailover policy has hardcoded #s of failovers and retries.  Just some 
more work to parse that.
-Needs lots of testing, been using iptables to block NNs on an HA cluster 
rather than actually switching active/standby.

> libhdfs++: HA namenode support
> --
>
> Key: HDFS-10441
> URL: https://issues.apache.org/jira/browse/HDFS-10441
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
> Attachments: HDFS-10441.HDFS-8707.000.patch
>
>
> If a cluster is HA enabled then do proper failover.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10402) DiskBalancer: Add QueryStatus command

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-10402:

Attachment: HDFS-10402-HDFS-1312.002.patch

Fixed the typo and rebased the patch against the top of the tree

> DiskBalancer: Add QueryStatus command
> -
>
> Key: HDFS-10402
> URL: https://issues.apache.org/jira/browse/HDFS-10402
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Affects Versions: HDFS-1312
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-1312
>
> Attachments: HDFS-10402-HDFS-1312.001.patch, 
> HDFS-10402-HDFS-1312.002.patch
>
>
> Adds QueryStatus which gets disk balancer status from a specific node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10441) libhdfs++: HA namenode support

[James Clampffer (JIRA)]

James Clampffer created HDFS-10441:
--

 Summary: libhdfs++: HA namenode support
 Key: HDFS-10441
 URL: https://issues.apache.org/jira/browse/HDFS-10441
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: James Clampffer


If a cluster is HA enabled then do proper failover.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10438) When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293639#comment-15293639
 ] 

Kihwal Lee commented on HDFS-10438:
---

Post-precommit +1 as the final nail on the coffin.

> When NameNode HA is configured to use the lifeline RPC server, it should log 
> the address of that server.
> 
>
> Key: HDFS-10438
> URL: https://issues.apache.org/jira/browse/HDFS-10438
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: ha, namenode
>Reporter: KWON BYUNGCHANG
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-10438.001.patch, HDFS-10438.002.patch
>
>
> As reported by [~magnum]:
> I have configured below
> {code}
> dfs.namenode.servicerpc-address.xdev.nn1=my.host.com:8040
> dfs.namenode.lifeline.rpc-address.xdev.nn1=my.host.com:8041
> {code}
> servicerpc port is 8040,  lifeline port is 8041.
> however zkfc daemon is logging using servicerpc port. 
> It may cause confusion.
> thank you.
> {code}
> 2016-05-19 19:18:40,566 WARN  ha.HealthMonitor 
> (HealthMonitor.java:doHealthChecks(207)) - Service health check failed for 
> NameNode at my.host.com/10.114.87.91:8040: The NameNode has no resources 
> available
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10440) Add more information to DataNode web UI

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293610#comment-15293610
 ] 

Kihwal Lee commented on HDFS-10440:
---

bq. Running state (active, decommissioning, decommissioned or lost etc)
These are only known to NN.  I wouldn't try to make it known to DN just for 
this.
It will be useful to display configured block pools along with each service 
actor's state (e.g. nnAddr, runningState, etc)
Showing per storage status will be most important.

> Add more information to DataNode web UI
> ---
>
> Key: HDFS-10440
> URL: https://issues.apache.org/jira/browse/HDFS-10440
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 2.5.0, 2.6.0, 2.7.0
>Reporter: Weiwei Yang
>
> At present, datanode web UI doesn't have much information except for node 
> name and port. Propose to add more information similar to namenode UI, 
> including, 
> * Static info (version, block pool  and cluster ID)
> * Running state (active, decommissioning, decommissioned or lost etc)
> * Summary (blocks, capacity, storage etc)
> * Utilities (logs)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10399) DiskBalancer: Add JMX for DiskBalancer

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-10399:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks for the review [~arpitagarwal]. I have committed this to the feature 
branch.

> DiskBalancer: Add JMX for DiskBalancer
> --
>
> Key: HDFS-10399
> URL: https://issues.apache.org/jira/browse/HDFS-10399
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Attachments: HDFS-10399-HDFS-1312.001.patch
>
>
> Expose diskbalancer status via JMX



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10439) Update setOwner doc in HdfsPermissionsGuide

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293484#comment-15293484
 ] 

John Zhuge edited comment on HDFS-10439 at 5/20/16 2:58 PM:


Patch 001:
* Update setOwner documentation regarding permission to change group


was (Author: jzhuge):
Patch 001:
* Update setOwner documentation regarding changing the group

> Update setOwner doc in HdfsPermissionsGuide
> ---
>
> Key: HDFS-10439
> URL: https://issues.apache.org/jira/browse/HDFS-10439
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>  Labels: supportability
> Attachments: HDFS-10439.001.patch
>
>
> In {{HdfsPermissionsGuide}}:
> bq. HDFS super-user access is not required to change the group, but the 
> caller must be a member of the specified group.
> should be
> bq. HDFS super-user access is not required to change the group, but the 
> caller must be the owner of the file and a member of the specified group.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10439) Update setOwner doc in HdfsPermissionsGuide

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HDFS-10439:
--
Attachment: HDFS-10439.001.patch

Patch 001:
* Update setOwner documentation regarding changing the group

> Update setOwner doc in HdfsPermissionsGuide
> ---
>
> Key: HDFS-10439
> URL: https://issues.apache.org/jira/browse/HDFS-10439
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>  Labels: supportability
> Attachments: HDFS-10439.001.patch
>
>
> In {{HdfsPermissionsGuide}}:
> bq. HDFS super-user access is not required to change the group, but the 
> caller must be a member of the specified group.
> should be
> bq. HDFS super-user access is not required to change the group, but the 
> caller must be the owner of the file and a member of the specified group.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10424) DatanodeLifelineProtocol not able to use under security cluster

[Vinayakumar B (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293476#comment-15293476
 ] 

Vinayakumar B commented on HDFS-10424:
--

Test failure seems unrelated.
+1 for the patch.



> DatanodeLifelineProtocol not able to use under security cluster
> ---
>
> Key: HDFS-10424
> URL: https://issues.apache.org/jira/browse/HDFS-10424
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: gu-chi
>Assignee: Chris Nauroth
>Priority: Blocker
> Attachments: HDFS-10424-branch-2.8.001.patch, HDFS-10424.001.patch
>
>
> {quote}
> protocol org.apache.hadoop.hdfs.server.protocol.DatanodeLifelineProtocol is 
> unauthorized for user * (auth:KERBEROS) | Server.java:1979
> {quote}
> I am using security cluster authenticate with kerberos, as I checked the the 
> code, if security auth enabled, because the DatanodeLifelineProtocol is not 
> inside HDFSPolicyProvider, when authorize in ServiceAuthorizationManager, 
> AuthorizationException will be thrown at line 96.
> Please point me out if I am wrong



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10424) DatanodeLifelineProtocol not able to use under security cluster

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293410#comment-15293410
 ] 

Hadoop QA commented on HDFS-10424:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 16s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
59s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 55s 
{color} | {color:green} branch-2.8 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 36s 
{color} | {color:green} branch-2.8 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
10s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 50s 
{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
32s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
39s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 2s 
{color} | {color:green} branch-2.8 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 51s 
{color} | {color:green} branch-2.8 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 31s 
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 31s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 27s 
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 27s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 5s 
{color} | {color:red} root: patch generated 4 new + 155 unchanged - 1 fixed = 
159 total (was 156) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 44s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 54s 
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 45s 
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 54s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_91. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 47m 56s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 17s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_101. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 45m 34s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
26s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} 

[jira] [Commented] (HDFS-10430) Refactor FileSystem#checkAccessPermissions for better reuse from tests

[Andras Bokor (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293342#comment-15293342
 ] 

Andras Bokor commented on HDFS-10430:
-

I got your point.
As far as I understand the {{InterfaceAudinece}} annotation is to inform 
callers if the method/class can/should be used from outside of the project.
If we add a public counterpart syntactically we do not break the 
{{InterfaceAudience}} rules but semantically we call a 
{{InterfaceAudience.Private}} method.
Please check HDFS-6570. As far as I understand 
{{FileSystem#checkAccessPermissions}} was made private due to some security 
issues.
Even if we add counterpart to a class we should consider using 
@VisibleForTesting annotation.

> Refactor FileSystem#checkAccessPermissions for better reuse from tests
> --
>
> Key: HDFS-10430
> URL: https://issues.apache.org/jira/browse/HDFS-10430
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> FileSystem#checkAccessPermissions could be used in a bunch of tests from 
> different projects, but it's in hadoop-common, which is not visible in some 
> cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10424) DatanodeLifelineProtocol not able to use under security cluster

[Vinayakumar B (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293187#comment-15293187
 ] 

Vinayakumar B commented on HDFS-10424:
--

Reports were not available to check the test failure.
bq. Slave went offline during the build

Re-triggered jenkins for this. 

> DatanodeLifelineProtocol not able to use under security cluster
> ---
>
> Key: HDFS-10424
> URL: https://issues.apache.org/jira/browse/HDFS-10424
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: gu-chi
>Assignee: Chris Nauroth
>Priority: Blocker
> Attachments: HDFS-10424-branch-2.8.001.patch, HDFS-10424.001.patch
>
>
> {quote}
> protocol org.apache.hadoop.hdfs.server.protocol.DatanodeLifelineProtocol is 
> unauthorized for user * (auth:KERBEROS) | Server.java:1979
> {quote}
> I am using security cluster authenticate with kerberos, as I checked the the 
> code, if security auth enabled, because the DatanodeLifelineProtocol is not 
> inside HDFSPolicyProvider, when authorize in ServiceAuthorizationManager, 
> AuthorizationException will be thrown at line 96.
> Please point me out if I am wrong



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8057) Move BlockReader implementation to the client implementation package

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293182#comment-15293182
 ] 

Hadoop QA commented on HDFS-8057:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 11s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 27 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 41s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 
1s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s 
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s 
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
46s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 27s 
{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
31s {color} | {color:green} branch-2 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s 
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in branch-2 has 7 
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s 
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 extant 
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s 
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 11s 
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s 
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 9s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s 
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s 
{color} | {color:red} hadoop-hdfs-project: patch generated 106 new + 699 
unchanged - 110 fixed = 805 total (was 809) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 18s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s 
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 5s 
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 49s 
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK 
v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 52m 0s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 59s 
{color} | {color:green} hadoop-hdfs-client in the patch 

[jira] [Commented] (HDFS-10424) DatanodeLifelineProtocol not able to use under security cluster

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293062#comment-15293062
 ] 

Hadoop QA commented on HDFS-10424:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m 11s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
44s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 11s 
{color} | {color:green} branch-2.8 passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 51s 
{color} | {color:green} branch-2.8 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
13s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 53s 
{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
36s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
47s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 6s 
{color} | {color:green} branch-2.8 passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 59s 
{color} | {color:green} branch-2.8 passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 56s 
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 56s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 48s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 8s 
{color} | {color:red} root: patch generated 4 new + 154 unchanged - 1 fixed = 
158 total (was 155) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 48s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
28s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 2s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s 
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 57s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 58s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_74. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 58s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 16s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 10s {color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
26s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | 

[jira] [Updated] (HDFS-8057) Move BlockReader implementation to the client implementation package

[Takanobu Asanuma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takanobu Asanuma updated HDFS-8057:
---
Attachment: HDFS-8057.branch-2.001.patch

I fixed the name of the patch.

> Move BlockReader implementation to the client implementation package
> 
>
> Key: HDFS-8057
> URL: https://issues.apache.org/jira/browse/HDFS-8057
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Takanobu Asanuma
> Attachments: HDFS-8057.1.patch, HDFS-8057.2.patch, HDFS-8057.3.patch, 
> HDFS-8057.4.patch, HDFS-8057.branch-2.001.patch, HDFS-8057.branch-2.5.patch
>
>
> BlockReaderLocal, RemoteBlockReader, etc should be moved to 
> org.apache.hadoop.hdfs.client.impl.  We may as well rename RemoteBlockReader 
> to BlockReaderRemote.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9806) Allow HDFS block replicas to be provided by an external storage system

[Thomas Demoor (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292968#comment-15292968
 ] 

Thomas Demoor commented on HDFS-9806:
-

Thanks [~chris.douglas] for the architecture doc. Very interesting feature.
 
First, the way we interpreted the document, the external (provided) storage is 
the source of truth so any changes there should be updated in HDFS and any 
inconsistencies that arise would favour the external store. With that in mind, 
we had some questions mostly relating to the following two paragraphs in 
section 3.4:
{quote}
Periodically, and/or when a particular directory or file is accessed on the 
Namenode, the Namenode queries the PROVIDED store to validate its cache. If the 
ID changed since its last update, the Namenode updates the corresponding 
metadata and block information.
The Datanode is also responsible for verifying the nonce when servicing read 
requests. Without this check, it may return data that does not match the record 
in the Namenode (e.g., if another file is renamed onto the same path in the 
external store).
{quote}
Questions:
# If the Namenode is accessing the PROVIDED storage to update its mapping 
shouldn’t it also update the nonce data at the same time and instruct the 
datanode to refresh too? Or is the intention for the Namenode to only update 
the directory information and not the actual nonce data for the files? (If so, 
how could the Namenode apply heuristics to detect “promoting output to a parent 
directory”?).
# How should this work in the face of Storage Policies? For example, if we have 
a StoragePolicy of {SSD, DISK, PROVIDED} it seems to us that it would make 
sense for the Namenode to use a HEAD request (or equivalent) to see if the data 
is still valid. If so, tell the client to talk to the Datanode with the file on 
SSD. Otherwise, the data needs to be refreshed across all three Datanodes. As 
the Namenode currently manages replication requests, it seems that it would 
make sense for it to trigger requests to refresh the data from the PROVIDED 
storage system.
# When you say “Periodically and/or when a particular directory or file is 
accessed on the Namenode” do you mean this is something to be configured, or 
just that it hasn’t been decided if both are required. We think periodically is 
required since this is the only way to clean up directory listings with files 
that have been removed from the PROVIDED storage. On access, it makes sense to 
always make a HEAD request (or equivalent) to make sure it isn’t stale.
# Finally, do you anticipate changes to the wire protocol between the Namenode 
and Datanode?


> Allow HDFS block replicas to be provided by an external storage system
> --
>
> Key: HDFS-9806
> URL: https://issues.apache.org/jira/browse/HDFS-9806
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Chris Douglas
> Attachments: HDFS-9806-design.001.pdf
>
>
> In addition to heterogeneous media, many applications work with heterogeneous 
> storage systems. The guarantees and semantics provided by these systems are 
> often similar, but not identical to those of 
> [HDFS|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/index.html].
>  Any client accessing multiple storage systems is responsible for reasoning 
> about each system independently, and must propagate/and renew credentials for 
> each store.
> Remote stores could be mounted under HDFS. Block locations could be mapped to 
> immutable file regions, opaque IDs, or other tokens that represent a 
> consistent view of the data. While correctness for arbitrary operations 
> requires careful coordination between stores, in practice we can provide 
> workable semantics with weaker guarantees.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10246) Standby NameNode dfshealth.jsp Response very slow

[zhangyubiao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292964#comment-15292964
 ] 

zhangyubiao commented on HDFS-10246:


[~kihwal]   thanks. it really helpful  for us. 

> Standby NameNode  dfshealth.jsp   Response very slow 
> -
>
> Key: HDFS-10246
> URL: https://issues.apache.org/jira/browse/HDFS-10246
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.2.0
> Environment: CentOS6.3  Hadoop-2.2.0 
>Reporter: zhangyubiao
>  Labels: bug
> Attachments: stacks.txt, 屏幕快照 2016-04-07 下午6.04.36.png
>
>
> HDFS Standby NameNode  dfshealth.jsp   Response very slow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10415) TestDistributedFileSystem#testDFSCloseOrdering() fails on branch-2

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292962#comment-15292962
 ] 

Steve Loughran commented on HDFS-10415:
---

actually, there's a good argument for having the configuration binding being 
done in an init() method; the Yarn Service model does the same thing: *it lets 
you subclass properly*. 

Any FS can subclass initialize(), patch up the config before calling the 
superclass, do something more after, all knowing that the rest of the 
overridden methods will get called. You cannot do things like that in 
constructors

> TestDistributedFileSystem#testDFSCloseOrdering() fails on branch-2
> --
>
> Key: HDFS-10415
> URL: https://issues.apache.org/jira/browse/HDFS-10415
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Affects Versions: 2.9.0
> Environment: jenkins
>Reporter: Sangjin Lee
>Assignee: Mingliang Liu
> Attachments: HDFS-10415-branch-2.000.patch, 
> HDFS-10415-branch-2.001.patch, HDFS-10415.000.patch
>
>
> {noformat}
> Tests run: 24, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 51.096 sec 
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestDistributedFileSystem
> testDFSCloseOrdering(org.apache.hadoop.hdfs.TestDistributedFileSystem)  Time 
> elapsed: 0.045 sec  <<< ERROR!
> java.lang.NullPointerException: null
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:790)
>   at 
> org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1417)
>   at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2084)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1187)
>   at 
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testDFSCloseOrdering(TestDistributedFileSystem.java:217)
> {noformat}
> This is with Java 8 on Mac. It passes fine on trunk. I haven't tried other 
> combinations. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8057) Move BlockReader implementation to the client implementation package

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292956#comment-15292956
 ] 

Hadoop QA commented on HDFS-8057:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s {color} 
| {color:red} HDFS-8057 does not apply to branch-2.5. Rebase required? Wrong 
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12805172/HDFS-8057.branch-2.5.patch
 |
| JIRA Issue | HDFS-8057 |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/15511/console |
| Powered by | Apache Yetus 0.2.0   http://yetus.apache.org |


This message was automatically generated.



> Move BlockReader implementation to the client implementation package
> 
>
> Key: HDFS-8057
> URL: https://issues.apache.org/jira/browse/HDFS-8057
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Takanobu Asanuma
> Attachments: HDFS-8057.1.patch, HDFS-8057.2.patch, HDFS-8057.3.patch, 
> HDFS-8057.4.patch, HDFS-8057.branch-2.5.patch
>
>
> BlockReaderLocal, RemoteBlockReader, etc should be moved to 
> org.apache.hadoop.hdfs.client.impl.  We may as well rename RemoteBlockReader 
> to BlockReaderRemote.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8057) Move BlockReader implementation to the client implementation package

[Takanobu Asanuma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takanobu Asanuma updated HDFS-8057:
---
Attachment: HDFS-8057.branch-2.5.patch

Hi, [~szetszwo].

Sorry for late reply. I uploaded a new patch for branch-2. Please review it 
again. Thanks.

> Move BlockReader implementation to the client implementation package
> 
>
> Key: HDFS-8057
> URL: https://issues.apache.org/jira/browse/HDFS-8057
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Takanobu Asanuma
> Attachments: HDFS-8057.1.patch, HDFS-8057.2.patch, HDFS-8057.3.patch, 
> HDFS-8057.4.patch, HDFS-8057.branch-2.5.patch
>
>
> BlockReaderLocal, RemoteBlockReader, etc should be moved to 
> org.apache.hadoop.hdfs.client.impl.  We may as well rename RemoteBlockReader 
> to BlockReaderRemote.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10438) When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292916#comment-15292916
 ] 

Hadoop QA commented on HDFS-10438:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 18s {color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
20s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 23s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestAsyncDFSRename |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12805149/HDFS-10438.002.patch |
| JIRA Issue | HDFS-10438 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 10ff22852954 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 9325845 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/15510/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HDFS-Build/15510/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/15510/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/15510/console |
| Powered by | Apache Yetus 0.2.0   http://yetus.apache.org |


This message was automatically generated.



> When NameNode HA is configured to use the lifeline RPC server, it should log 
> the address of that server.
> 
>
> Key: HDFS-10438
> URL: 

[jira] [Created] (HDFS-10440) Add more information to DataNode web UI

[Weiwei Yang (JIRA)]

Weiwei Yang created HDFS-10440:
--

 Summary: Add more information to DataNode web UI
 Key: HDFS-10440
 URL: https://issues.apache.org/jira/browse/HDFS-10440
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: datanode
Affects Versions: 2.7.0, 2.6.0, 2.5.0
Reporter: Weiwei Yang


At present, datanode web UI doesn't have much information except for node name 
and port. Propose to add more information similar to namenode UI, including, 

* Static info (version, block pool  and cluster ID)
* Running state (active, decommissioning, decommissioned or lost etc)
* Summary (blocks, capacity, storage etc)
* Utilities (logs)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292808#comment-15292808
 ] 

Hadoop QA commented on HDFS-10420:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 11 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 
12s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s 
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
32s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s 
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s 
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s 
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 57s {color} 
| {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 115 unchanged - 
1 fixed = 116 total (was 116) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
30s {color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: patch generated 0 
new + 27 unchanged - 16 fixed = 27 total (was 43) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 35s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 0s {color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
25s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 40s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.TestOzoneRestWithMiniCluster |
|   | hadoop.hdfs.server.datanode.TestBlockPoolManager |
|   | hadoop.hdfs.server.balancer.TestBalancer |
|   | hadoop.hdfs.TestAsyncDFSRename |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12805134/HDFS-10420-HDFS-7240.05.patch
 |
| JIRA Issue | HDFS-10420 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux d72fa4959772 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | HDFS-7240 / 76279f5 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| javac | hadoop-hdfs-project_hadoop-hdfs: 
https://builds.apache.org/job/PreCommit-HDFS-Build/15508/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/15508/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HDFS-Build/15508/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |

[jira] [Commented] (HDFS-10420) Fix Ozone unit tests to use MiniOzoneCluster

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292789#comment-15292789
 ] 

Hadoop QA commented on HDFS-10420:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 11 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 
8s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s 
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
30s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s 
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s 
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s 
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 26s {color} 
| {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 115 unchanged - 
1 fixed = 116 total (was 116) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
24s {color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: patch generated 0 
new + 27 unchanged - 16 fixed = 27 total (was 43) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 33s {color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
29s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m 48s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.TestOzoneRestWithMiniCluster |
|   | hadoop.hdfs.TestAsyncDFSRename |
|   | hadoop.hdfs.server.datanode.TestBlockPoolManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12805134/HDFS-10420-HDFS-7240.05.patch
 |
| JIRA Issue | HDFS-10420 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 3835db93708e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | HDFS-7240 / 76279f5 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| javac | hadoop-hdfs-project_hadoop-hdfs: 
https://builds.apache.org/job/PreCommit-HDFS-Build/15506/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/15506/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HDFS-Build/15506/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 

[jira] [Commented] (HDFS-10436) dfs.block.access.token.enable should default on when security is !simple

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292787#comment-15292787
 ] 

Hadoop QA commented on HDFS-10436:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 26s {color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
19s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 57s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestCrcCorruption |
|   | hadoop.hdfs.TestFileChecksum |
|   | hadoop.hdfs.server.datanode.TestTransferRbw |
|   | hadoop.hdfs.TestDataTransferProtocol |
|   | hadoop.hdfs.TestAsyncDFSRename |
|   | hadoop.hdfs.server.datanode.TestBlockReplacement |
|   | hadoop.hdfs.server.namenode.TestFsck |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12805128/HDFS-10436.001.patch |
| JIRA Issue | HDFS-10436 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 0dac96d0af30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 21d2b90 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/15507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HDFS-Build/15507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/15507/testReport/ |
| modules | C: