[jira] [Updated] (HDFS-3562) Handle disconnect and session timeout events at BKJM

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HDFS-3562:
---
Resolution: Won't Fix
Status: Resolved  (was: Patch Available)

Closing as wn't fix as per discussion 
[here|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3c1288296033.5942327.1469727453010.javamail.ya...@mail.yahoo.com%3E]
 and even not require after HDFS-10957 

> Handle disconnect and session timeout events at BKJM
> 
>
> Key: HDFS-3562
> URL: https://issues.apache.org/jira/browse/HDFS-3562
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>  Labels: BB2015-05-TBR
> Attachments: HDFS-3562.patch, HDFS-3562.patch
>
>
> # Retry zookeeper operations for some amount of time in case of 
> CONNECTIONLOSS/OPERATIONTIMEOUT exceptions.
> # In case of Session expiry trigger shutdown



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-3908) In HA mode, when there is a ledger in BK missing, which is generated after the last checkpoint, NN can not restore it.

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula resolved HDFS-3908.

Resolution: Won't Fix

Closing as wn't fix as per discussion 
[here|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3c1288296033.5942327.1469727453010.javamail.ya...@mail.yahoo.com%3E]
 and even not require after HDFS-10957 

> In HA mode, when there is a ledger in BK missing, which is generated after 
> the last checkpoint, NN can not restore it.
> --
>
> Key: HDFS-3908
> URL: https://issues.apache.org/jira/browse/HDFS-3908
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Affects Versions: 2.0.1-alpha
>Reporter: Han Xiao
>
> If not HA, when the num of edits.dir is larger than 1. Missing of one editlog 
> file in a dir will not relust problem cause of the replica in the other dir. 
> However, when in HA mode(using BK as ShareStorage), if an ledger missing, the 
> missing ledger will not restored at the phase of NN starting even if the 
> related editlog file existing in local dir.
> The missing maintains when NN is still in standby state. However, when the NN 
> enters active state, it will read the editlog file(related to the missing 
> ledger) in local. But, unfortunately, the ledger after the missing one in BK 
> can't be readed at such a phase(cause of gap).
> Therefore in the following situation, editlogs will not be restored even 
> there is an editlog file either in BK or in local dir: 
> In such a stituation, editlog can't be restored:
> 1、fsiamge file: fsimage_0005946.md5
> 2、legder in zk:
>   \[zk: localhost:2181(CONNECTED) 0\] ls 
> /hdfsEdit/ledgers/edits_00594
>   edits_005941_005942
>   edits_005943_005944
>   edits_005945_005946
>   edits_005949_005949   
> （missing edits_005947_005948）
> 3、editlog in local editlog dir：
>   \-rw-r--r-- 1 root root  30 Sep  8 03:24 
> edits_0005947-0005948
>   \-rw-r--r-- 1 root root 1048576 Sep  8 03:35 
> edits_0005950-0005950
>   \-rw-r--r-- 1 root root 1048576 Sep  8 04:42 
> edits_0005951-0005951
>   （miss edits_0005949-0005919）
> 4、and the seen_txid
>   vm2:/tmp/hadoop-root/dfs/name/current # cat seen_txid
>   5949
> Here, we want to restored editlog from txid 5946(image) to txid 
> 5949(seen_txid). The 5947-5948 is missing in BK, 5949-5949 is missing in 
> local dir.
> When start the NN, the following exception is thrown:
> 2012-09-08 06:26:10,031 FATAL 
> org.apache.hadoop.hdfs.server.namenode.NameNode: Error encountered requiring 
> NN shutdown. Shutting down immediately.
> java.io.IOException: There appears to be a gap in the edit log.  We expected 
> txid 5949, but got txid 5950.
> at 
> org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:163)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:93)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:692)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:223)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.catchupDuringFailover(EditLogTailer.java:182)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:599)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1325)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.HAState.setStateInternal(HAState.java:63)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.setState(StandbyState.java:49)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.transitionToActive(NameNode.java:1233)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.transitionToActive(NameNodeRpcServer.java:990)
> at 
> org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.transitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107)
> at 
> org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:3633)
> at 
> 

[jira] [Resolved] (HDFS-4613) Improve BKJM documentation

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula resolved HDFS-4613.

Resolution: Won't Fix

Closing as wn't fix as per discussion 
[here|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3c1288296033.5942327.1469727453010.javamail.ya...@mail.yahoo.com%3E]
 and even not require after HDFS-10957 

> Improve BKJM documentation
> --
>
> Key: HDFS-4613
> URL: https://issues.apache.org/jira/browse/HDFS-4613
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Flavio Junqueira
>Assignee: Rakesh R
>Priority: Minor
>
> The documentation tags BKJM as experimental, and I don't think it is 
> experimental any longer. It is also missing a link of HDFS HA with BKJM on 
> the left-hand side menu of the documentation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-3399) BookKeeper option support for NN HA

[Brahma Reddy Battula (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554191#comment-15554191
 ] 

Brahma Reddy Battula commented on HDFS-3399:


Going to close all the subtasks as per discussion 
[here|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3c1288296033.5942327.1469727453010.javamail.ya...@mail.yahoo.com%3E]
 and not required after HDFS-10957.

> BookKeeper option support for NN HA
> ---
>
> Key: HDFS-3399
> URL: https://issues.apache.org/jira/browse/HDFS-3399
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: ha
>Affects Versions: 2.0.0-alpha, 3.0.0-alpha1
>Reporter: Uma Maheswara Rao G
> Attachments: BKTestDoc.pdf
>
>
> Here is the JIRA to BookKeeper support issues with NN HA. We can file all the 
> BookKeeperJournalManager issues under this JIRA for more easy tracking.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10977) Balancer should query NameNode with a timeout

[Senthilkumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Senthilkumar updated HDFS-10977:

Attachment: Gmail - HDFS Balancer Stuck after 10 Minz.pdf

> Balancer should query NameNode with a timeout
> -
>
> Key: HDFS-10977
> URL: https://issues.apache.org/jira/browse/HDFS-10977
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: Gmail - HDFS Balancer Stuck after 10 Minz.pdf, 
> HDFS-10977-reproduce.patch
>
>
> We found a case where {{Dispatcher}} was stuck at {{getBlockList}} *forever* 
> (well, several hours when we found it).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10977) Balancer should query NameNode with a timeout

[Senthilkumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554136#comment-15554136
 ] 

Senthilkumar commented on HDFS-10977:
-

[~zhz] ,  When i was working with balancer last month i too faced this issue ,

2016-09-08 06:32:06,574 DEBUG org.apache.hadoop.ipc.Client: IPC Client 
(685788708) connection to host/10.103.108. 
201:8020 from hadoop/host@DOMAIN sending #49230 
2016-09-08 06:32:06,574 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: 
getBlocks took 0ms 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client 
(685788708) connection to host/10.103.108. 
201:8020 from hadoop/host@DOMAIN sending #49231 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client 
(685788708) connection to host/10.103.108. 
201:8020 from hadoop/host@DOMAIN got value #49229 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: 
getBlocks took 1ms 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client 
(685788708) connection to host/10.103.108. 
201:8020 from hadoop/host@DOMAIN got value #49230 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client 
(685788708) connection to host/10.103.108. 
201:8020 from hadoop/host@DOMAIN sending #49232 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: 
getBlocks took 1ms 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client 
(685788708) connection to host/10.103.108. 
201:8020 from hadoop/host@DOMAIN sending #49233 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client 
(685788708) connection to host/10.103.108. 
201:8020 from hadoop/host@DOMAIN got value #49231 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: 
getBlocks took 0ms 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client 
(685788708) connection to host/10.103.108. 
201:8020 from hadoop/host@DOMAIN sending #49234 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client 
(685788708) connection to host/10.103.108. 
201:8020 from hadoop/host@DOMAIN got value #49232 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: 
getBlocks took 0ms 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client 
(685788708) connection to host/10.103.108. 
201:8020 from hadoop/host@DOMAIN sending #49235 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client 
(685788708) connection to host/10.103.108. 
201:8020 from hadoop/host@DOMAIN got value #49233 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: 
getBlocks took 0ms 
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client 
(685788708) connection to host/10.103.108. 
201:8020 from hadoop/host@DOMAIN got value #49234


Here is the TRACE where it took few hours and no blocks found .. I ended up in 
Restarting balancer to make it work..


At last i started balancer with -include option ( by pulling LIVE DNs ) and it 
helped .. Not sure including live nodes is the right solution ?? .. If it is 
the case i think problem is balancer taking decomm/decommed nodes as well ??

Attaching the discussion which i started sometime back ..


> Balancer should query NameNode with a timeout
> -
>
> Key: HDFS-10977
> URL: https://issues.apache.org/jira/browse/HDFS-10977
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10977-reproduce.patch
>
>
> We found a case where {{Dispatcher}} was stuck at {{getBlockList}} *forever* 
> (well, several hours when we found it).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10973) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554035#comment-15554035
 ] 

Xiaobing Zhou commented on HDFS-10973:
--

Thanks [~brahmareddy], you are correct, I just noticed it's been created twice.

> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10973
> URL: https://issues.apache.org/jira/browse/HDFS-10973
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10975) fsck -list-corruptfileblocks does not report corrupt EC files

[Takanobu Asanuma (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553930#comment-15553930
 ] 

Takanobu Asanuma commented on HDFS-10975:
-

[~jojochuang]
Thank you for filing the issue! If you haven't started working on this jira, 
can I take it?

> fsck -list-corruptfileblocks does not report corrupt EC files
> -
>
> Key: HDFS-10975
> URL: https://issues.apache.org/jira/browse/HDFS-10975
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>
> HDFS-10826 fix fsck for corrupt EC files if no parameters are specified.
> However, if I change the test case added in HDFS-10826 
> (TestFsck#testFsckCorruptECFile) to run "fsck -list-corruptfileblocks", the 
> same test test failed because fsck reports no corrupt files. 
> Interestingly, if I run "fsck -files -blocks -replicaDetails" then the test 
> passed and shows the corrupt file.
> Need to fix the discrepancy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-10973) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula resolved HDFS-10973.
-
Resolution: Duplicate

[~xiaobingo] resolving as duplicate of HDFS-10972..Please feel re-open,if it's 
not duplicate.

> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10973
> URL: https://issues.apache.org/jira/browse/HDFS-10973
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-10974) Document replication factor for EC files.

[Yiqun Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin reassigned HDFS-10974:


Assignee: Yiqun Lin

> Document replication factor for EC files.
> -
>
> Key: HDFS-10974
> URL: https://issues.apache.org/jira/browse/HDFS-10974
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
>
> The replication factor of an EC file is 0 from a user perspective. This 
> affects all user visible commands and APIs, such as {{hdfs dfs -ls}}, {{hdfs 
> dfs -stat}}, webhdfs, httpfs...etc.
> In addition, FileSystem#setReplication API for an EC file is ignored. So is 
> the command {{hdfs dfs -setrep}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553665#comment-15553665
 ] 

Mingliang Liu commented on HDFS-10903:
--

Test failure {{hadoop.tools.TestHdfsConfigFields}} is related. We can fix this 
either by documenting it in config xml files, or adding it to 
{{TestHdfsConfigFields#configurationPropsToSkipCompare}}. I prefer the former 
one. [~ajisakaa] any thoughts? Thanks.

> Replace config key literal strings with config key names II: hadoop hdfs
> 
>
> Key: HDFS-10903
> URL: https://issues.apache.org/jira/browse/HDFS-10903
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mingliang Liu
>Assignee: Chen Liang
>Priority: Minor
> Attachments: HADOOP-13644.001.patch, HDFS-10903.002.patch
>
>
> In *Hadoop HDFS*, there are some places that use config key literal strings 
> instead of config key names, e.g.
> {code:title=IOUtils.java}
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
> {code}
> We should replace places like this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553642#comment-15553642
 ] 

Hadoop QA commented on HDFS-10903:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
52s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
32s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 487 
unchanged - 1 fixed = 487 total (was 488) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m  0s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
48s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 20s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10903 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832014/HDFS-10903.002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux bdf59385b589 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / bf37217 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17054/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17054/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs 

[jira] [Updated] (HDFS-10977) Balancer should query NameNode with a timeout

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-10977:
-
Attachment: HDFS-10977-reproduce.patch

Uploading a patch to demonstrate the issue (not the solution). Log from running 
the test:
{code}
2016-10-06 16:27:19,627 [DataXceiver for client 
DFSClient_NONMAPREDUCE_-1700201955_1 at /127.0.0.1:54189 [Receiving block 
BP-1747605867-172.21.144.175-1475796435325:blk_1073741825_1001]] INFO  
datanode.DataNode (DataXceiver.java:writeBlock(705)) - Receiving 
BP-1747605867-172.21.144.175-1475796435325:blk_1073741825_1001 src: 
/127.0.0.1:54189 dest: /127.0.0.1:54178
2016-10-06 16:27:19,654 [DataXceiver for client 
DFSClient_NONMAPREDUCE_-1700201955_1 at /127.0.0.1:54190 [Receiving block 
BP-1747605867-172.21.144.175-1475796435325:blk_1073741825_1001]] INFO  
datanode.DataNode (DataXceiver.java:writeBlock(705)) - Receiving 
BP-1747605867-172.21.144.175-1475796435325:blk_1073741825_1001 src: 
/127.0.0.1:54190 dest: /127.0.0.1:54183
2016-10-06 16:27:19,660 [DataXceiver for client 
DFSClient_NONMAPREDUCE_-1700201955_1 at /127.0.0.1:54191 [Receiving block 
BP-1747605867-172.21.144.175-1475796435325:blk_1073741825_1001]] INFO  
datanode.DataNode (DataXceiver.java:writeBlock(705)) - Receiving 
BP-1747605867-172.21.144.175-1475796435325:blk_1073741825_1001 src: 
/127.0.0.1:54191 dest: /127.0.0.1:54174
2016-10-06 16:27:19,703 [IPC Server handler 2 on 54173] INFO  hdfs.StateChange 
(FSNamesystem.java:fsync(3027)) - BLOCK* fsync: /system/balancer.id for 
DFSClient_NONMAPREDUCE_-1700201955_1
2016-10-06 16:32:19,760 [IPC Server handler 5 on 54173] WARN  BlockStateChange 
(BlockManager.java:getBlocksWithLocations(1269)) - BLOCK* getBlocks: Asking for 
blocks from an unrecorded node null:0
{code}

So the connector waited 5 mins.

> Balancer should query NameNode with a timeout
> -
>
> Key: HDFS-10977
> URL: https://issues.apache.org/jira/browse/HDFS-10977
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10977-reproduce.patch
>
>
> We found a case where {{Dispatcher}} was stuck at {{getBlockList}} *forever* 
> (well, several hours when we found it).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10977) Balancer should query NameNode with a timeout

[Zhe Zhang (JIRA)]

Zhe Zhang created HDFS-10977:


 Summary: Balancer should query NameNode with a timeout
 Key: HDFS-10977
 URL: https://issues.apache.org/jira/browse/HDFS-10977
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer & mover
Reporter: Zhe Zhang
Assignee: Zhe Zhang


We found a case where {{Dispatcher}} was stuck at {{getBlockList}} *forever* 
(well, several hours when we found it).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10976) Fsck should mark EC files explicitly

[Wei-Chiu Chuang (JIRA)]

Wei-Chiu Chuang created HDFS-10976:
--

 Summary: Fsck should mark EC files explicitly
 Key: HDFS-10976
 URL: https://issues.apache.org/jira/browse/HDFS-10976
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: 3.0.0-alpha2
Reporter: Wei-Chiu Chuang


Currently fsck reports corrupt EC files as follows: it does not distinguish 
erasure coded files versus replicated files. In addition, it would be nice to 
print out the ec policy of the corrupt EC file.

{quote}
/striped 
/striped/corrupted 393216 bytes, 1 block(s): 
/striped/corrupted: CORRUPT blockpool BP-1564681138-127.0.0.1-1475793860787 
block blk_-9223372036854775792
 Under replicated 
BP-1564681138-127.0.0.1-1475793860787:blk_-9223372036854775792_1001. Target 
Replicas is 9 but found 5 live replica(s), 0 decommissioned replica(s) and 0 
decommissioning replica(s).
 CORRUPT 1 blocks of total size 393216 B
0. BP-1564681138-127.0.0.1-1475793860787:blk_-9223372036854775792_1001 
len=393216 Live_repl=5  
[DatanodeInfoWithStorage[127.0.0.1:62192,DS-81dcbc38-755e-446e-a028-71a79e4de6d9,DISK](CORRUPT),
 
DatanodeInfoWithStorage[127.0.0.1:62180,DS-98fe193d-6342-4b2c-ad61-4586e2530b1e,DISK](CORRUPT),
 
DatanodeInfoWithStorage[127.0.0.1:62167,DS-53031f88-0c63-4839-ab18-efea2f1bb063,DISK](CORRUPT),
 
DatanodeInfoWithStorage[127.0.0.1:62162,DS-e8b418fd-165d-4d6f-886b-f75c21be096d,DISK](CORRUPT),
 
DatanodeInfoWithStorage[127.0.0.1:62176,DS-03e51584-5b33-4bb6-89b5-f519cda57429,DISK](LIVE),
 
DatanodeInfoWithStorage[127.0.0.1:62158,DS-ce9ca7b3-5b00-4351-8537-822eed532b46,DISK](LIVE),
 
DatanodeInfoWithStorage[127.0.0.1:62184,DS-668c500e-eb3d-4d40-b900-814076d5e160,DISK](LIVE),
 
DatanodeInfoWithStorage[127.0.0.1:62171,DS-763a3961-b214-4601-81c0-abdaecf539c4,DISK](LIVE),
 
DatanodeInfoWithStorage[127.0.0.1:62188,DS-d4ea6399-bead-452e-8dd5-bc8c5ebd4f45,DISK](LIVE)]

{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10912) Ozone:SCM: Add chill mode support to NodeManager.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553516#comment-15553516
 ] 

Hadoop QA commented on HDFS-10912:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated 
0 new + 5 unchanged - 1 fixed = 5 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 31s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 20s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.TestOzoneRestWithMiniCluster |
|   | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
|   | hadoop.hdfs.server.datanode.TestBlockPoolManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10912 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832044/HDFS-10912-HDFS-7240.003.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 76c3b16fa9ec 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | HDFS-7240 / 4217f85 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17052/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17052/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17052/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Ozone:SCM: Add chill mode support to NodeManager.
> -
>
> Key: HDFS-10912
> URL: 

[jira] [Commented] (HDFS-10975) fsck -list-corruptfileblocks does not report corrupt EC files

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553483#comment-15553483
 ] 

Wei-Chiu Chuang commented on HDFS-10975:


It is also worth noting that without any parameters, the corrupt EC file is 
also reported as having an underr replicated block. An EC block should not be 
called under replicated if it misses 4 "replicas".

> fsck -list-corruptfileblocks does not report corrupt EC files
> -
>
> Key: HDFS-10975
> URL: https://issues.apache.org/jira/browse/HDFS-10975
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>
> HDFS-10826 fix fsck for corrupt EC files if no parameters are specified.
> However, if I change the test case added in HDFS-10826 
> (TestFsck#testFsckCorruptECFile) to run "fsck -list-corruptfileblocks", the 
> same test test failed because fsck reports no corrupt files. 
> Interestingly, if I run "fsck -files -blocks -replicaDetails" then the test 
> passed and shows the corrupt file.
> Need to fix the discrepancy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10975) fsck -list-corruptfileblocks does not report corrupt EC files

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-10975:
---
Issue Type: Sub-task  (was: Bug)
Parent: HDFS-8031

> fsck -list-corruptfileblocks does not report corrupt EC files
> -
>
> Key: HDFS-10975
> URL: https://issues.apache.org/jira/browse/HDFS-10975
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>
> HDFS-10826 fix fsck for corrupt EC files if no parameters are specified.
> However, if I change the test case added in HDFS-10826 
> (TestFsck#testFsckCorruptECFile) to run "fsck -list-corruptfileblocks", the 
> same test test failed because fsck reports no corrupt files. 
> Interestingly, if I run "fsck -files -blocks -replicaDetails" then the test 
> passed and shows the corrupt file.
> Need to fix the discrepancy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553474#comment-15553474
 ] 

Hadoop QA commented on HDFS-10972:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 26s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 12 new + 201 unchanged - 0 fixed = 213 total (was 201) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 60m  
1s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 52s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10972 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832039/HDFS-10972.000.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux aab8ec5cfd9d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 48b9d5f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17051/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17051/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17051/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10972
> URL: https://issues.apache.org/jira/browse/HDFS-10972
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: 

[jira] [Commented] (HDFS-10801) [SPS]: Protocol buffer changes for sending storage movement commands from NN to DN

[Uma Maheswara Rao G (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553458#comment-15553458
 ] 

Uma Maheswara Rao G commented on HDFS-10801:


Thank you [~rakeshr] for working on this.
Overall patch looks good to me. I have few minor comments though.

* Code piece 
{code}
+message BlockStorageMovementProto {
+  required BlockProto block = 1;
+  required DatanodeInfosProto sourceDnInfos = 2;
+  required DatanodeInfosProto targetDnInfos = 3;
+  required StorageTypesProto sourceStorageTypes = 4;
+  required StorageTypesProto targetStorageTypes = 5;
+}
{code}
Can you please check the variable ordering consistent? BockMovingInfo toString 
also has different ordering. It would be good if we follow consistent ordering.

* BlockStorageMovementCommand: I would like to see javadoc for the variables 
and methods here. Explaining a bit about trackID makes more clear.

* processBlockMovingTasks: I would like to see adding comment by drafting 
overall processing idea here. I am fine if you want to add in other place, but 
you can refer link


> [SPS]: Protocol buffer changes for sending storage movement commands from NN 
> to DN 
> ---
>
> Key: HDFS-10801
> URL: https://issues.apache.org/jira/browse/HDFS-10801
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Affects Versions: HDFS-10285
>Reporter: Uma Maheswara Rao G
>Assignee: Rakesh R
> Attachments: HDFS-10801-HDFS-10285-00.patch
>
>
> This JIRA is for tracking the work of protocol buffer changes for sending the 
> storage movement commands from NN to DN



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553439#comment-15553439
 ] 

Mingliang Liu commented on HDFS-10972:
--

I found most of my comments can be addressed if you refer the tests in 
{{TestDFSAdminWithHA.java}}. Thanks.

> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10972
> URL: https://issues.apache.org/jira/browse/HDFS-10972
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10972.000.patch
>
>
> getDatanodeInfo should be tested in admin CLI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-5684) o.a.h.fs.viewfs.ViewFileSystem.MountPoint has no public methods.

[Manoj Govindassamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manoj Govindassamy updated HDFS-5684:
-
Affects Version/s: (was: 2.2.0)
   3.0.0-alpha2
 Target Version/s: 3.0.0-alpha2
   Status: Patch Available  (was: Open)

> o.a.h.fs.viewfs.ViewFileSystem.MountPoint has no public methods.
> 
>
> Key: HDFS-5684
> URL: https://issues.apache.org/jira/browse/HDFS-5684
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Keith Turner
>Assignee: Manoj Govindassamy
> Attachments: HDFS-5684.01.patch
>
>
> o.a.h.fs.viewfs.ViewFileSystem provides the following method
> {noformat}
>   public MountPoint[] getMountPoints()
> {noformat}
> However MountPoint has no public methods making getMountPoints() useless.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen Liang updated HDFS-10903:
--
Status: Patch Available  (was: Open)

> Replace config key literal strings with config key names II: hadoop hdfs
> 
>
> Key: HDFS-10903
> URL: https://issues.apache.org/jira/browse/HDFS-10903
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mingliang Liu
>Assignee: Chen Liang
>Priority: Minor
> Attachments: HADOOP-13644.001.patch, HDFS-10903.002.patch
>
>
> In *Hadoop HDFS*, there are some places that use config key literal strings 
> instead of config key names, e.g.
> {code:title=IOUtils.java}
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
> {code}
> We should replace places like this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-5684) o.a.h.fs.viewfs.ViewFileSystem.MountPoint has no public methods.

[Manoj Govindassamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manoj Govindassamy updated HDFS-5684:
-
Attachment: HDFS-5684.01.patch

Attaching v01 patch to expose src and targets from 
{{ViewFileSystem#MountPoint}}. Updated a test to make use of these methods. 
These public methods are more useful only if there are users from other 
packages. 

> o.a.h.fs.viewfs.ViewFileSystem.MountPoint has no public methods.
> 
>
> Key: HDFS-5684
> URL: https://issues.apache.org/jira/browse/HDFS-5684
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs-client
>Affects Versions: 2.2.0
>Reporter: Keith Turner
>Assignee: Manoj Govindassamy
> Attachments: HDFS-5684.01.patch
>
>
> o.a.h.fs.viewfs.ViewFileSystem provides the following method
> {noformat}
>   public MountPoint[] getMountPoints()
> {noformat}
> However MountPoint has no public methods making getMountPoints() useless.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10967) Add configuration for BlockPlacementPolicy to avoid near-full DataNodes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553428#comment-15553428
 ] 

Hadoop QA commented on HDFS-10967:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 28s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 3 new + 433 unchanged - 0 fixed = 436 total (was 433) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 12s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 33s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10967 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832034/HDFS-10967.poc.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 39bd620beae5 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 1d330fb |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17050/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17050/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17050/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17050/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add configuration for BlockPlacementPolicy to avoid 

[jira] [Updated] (HDFS-10893) Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HDFS-10893:
-
Fix Version/s: 3.0.0-alpha2

> Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test
> 
>
> Key: HDFS-10893
> URL: https://issues.apache.org/jira/browse/HDFS-10893
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: HDFS-10893-branch-2.000.patch, 
> HDFS-10893-branch-2.001.patch, HDFS-10893-branch-2.002.patch, 
> HDFS-10893-branch-2.002.patch, HDFS-10893.000.patch, HDFS-10893.001.patch, 
> HDFS-10893.002.patch
>
>
> It seems that setting up MiniDFSCluser once for all commands test will reduce 
> the total time. To share a global cluster, the tests should use individual 
> test directories to avoid conflict between test cases. Meanwhile, the 
> MiniDFSCluster should not use the default root data directory; or else tests 
> are not able to launch another cluster(s) by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10965) Add unit test for HDFS command 'dfsadmin -printTopology'

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553400#comment-15553400
 ] 

Mingliang Liu commented on HDFS-10965:
--

Thanks for the patch, [~xiaobingo]. Can you update the patch to 
address/consider the common comments I made in [HDFS-10972]? Thanks.

> Add unit test for HDFS command 'dfsadmin -printTopology'
> 
>
> Key: HDFS-10965
> URL: https://issues.apache.org/jira/browse/HDFS-10965
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10965.000.patch, HDFS-10965.001.patch, 
> HDFS-10965.002.patch, HDFS-10965.003.patch
>
>
> DFSAdmin#printTopology should also be tested. This proposes adding it in 
> TestDFSAdmin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553388#comment-15553388
 ] 

Mingliang Liu commented on HDFS-10972:
--

Thanks for the patch. I think this UT is useful. I'll confirm that the command 
is not covered yet by the current tests.

# Adding support to redirect System.out in {{DFSAdmin}} per se is not preferred.
{code}
final DFSAdmin dfsAdmin = new DFSAdmin(dfsConf, out, err);
{code}
If your goal is to test the System.out output message, you can use 
{{System.setOut()}}. Code snippet like:
{code}
final PrintStream stdout = System.out;
try {
  final PrintStream out = new PrintStream();
  System.setOut(out);
  // ...
} finally {
  System.setOut(stdout);
}
{code}
# timeout is good for unit tests.
{code}
142   //@Test(timeout = 3)
143   @Test
{code}
# {{scanIntoList()}} considered static?
# {{miniCluster}} can be used in try-with-resource style. Code snippet as:
{code}
try (MiniDFSCluster cluster = new 
MiniDFSCluster.Builder(dfsConf).numDataNodes(numDn).build()) {
  cluster.waitForActive();
  // ...
}
{code}
# I don't like the coding style (IMHO, shorter is better, if not losing 
readability). Can you confirm that it adheres to the Hadoop coding guide?
{code}
170 /* invoke getDatanodeInfo */
171 final int ret = ToolRunner.run(
172 dfsAdmin,
173 new String[] {
174 "-getDatanodeInfo",
175 String.format(
176 "%s:%d",
177 dn.getXferAddress().getHostString(),
178 dn.getIpcPort()) });
179 
{code}
# Can you add failing message to the assertion {{assertEquals(1, 
outs.size());}} as it's not straightforward why the size is 1 (one line per DN 
I guess?)

> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10972
> URL: https://issues.apache.org/jira/browse/HDFS-10972
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10972.000.patch
>
>
> getDatanodeInfo should be tested in admin CLI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10925) Cache symlinkString in INodeSymlink

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553377#comment-15553377
 ] 

Kihwal Lee commented on HDFS-10925:
---

[~daryn] Please have a look.

> Cache symlinkString in INodeSymlink
> ---
>
> Key: HDFS-10925
> URL: https://issues.apache.org/jira/browse/HDFS-10925
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10925.001.patch, HDFS-10925.002.patch, 
> HDFS-10925.003.patch
>
>
> In {{INodeSymlink}}'s construct method, it will transfer the input symlink 
> string to a byte array. If we want to invoke 
> {{INodeSymlink#getSymlinkString}}, it will transfer the byte array to the 
> string again. Since we don't cache symlinkString  here, it will do the 
> {{DFSUtil.bytes2String}} method every time. It seems not efficient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10826) Correctly report missing EC blocks in FSCK

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-10826:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
   Status: Resolved  (was: Patch Available)

> Correctly report missing EC blocks in FSCK
> --
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10826.2.patch, HDFS-10826.3.patch, 
> HDFS-10826.4.patch, HDFS-10826.5.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
>  Total size:536870912 B
>  Total files:   1
>  Total block groups (validated):1 (avg. block group size 536870912 B)
>   
>   UNRECOVERABLE BLOCK GROUPS:   1 (100.0 %)
>   
>  Minimally erasure-coded block groups:  0 (0.0 %)
>  Over-erasure-coded block groups:   0 (0.0 %)
>  Under-erasure-coded block groups:  1 (100.0 %)
>  Unsatisfactory placement block groups: 0 (0.0 %)
>  Default ecPolicy:  RS-DEFAULT-6-3-64k
>  Average block group size:  5.0
>  Missing block groups:  0
>  Corrupt block groups:  0
>  Missing internal blocks:   4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL. 
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is 
> unrecoverable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10826) Correctly report missing EC blocks in FSCK

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553336#comment-15553336
 ] 

Wei-Chiu Chuang commented on HDFS-10826:


The code seems to have bug. Please see HDFS-10975 for discussion.

> Correctly report missing EC blocks in FSCK
> --
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.3.patch, 
> HDFS-10826.4.patch, HDFS-10826.5.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
>  Total size:536870912 B
>  Total files:   1
>  Total block groups (validated):1 (avg. block group size 536870912 B)
>   
>   UNRECOVERABLE BLOCK GROUPS:   1 (100.0 %)
>   
>  Minimally erasure-coded block groups:  0 (0.0 %)
>  Over-erasure-coded block groups:   0 (0.0 %)
>  Under-erasure-coded block groups:  1 (100.0 %)
>  Unsatisfactory placement block groups: 0 (0.0 %)
>  Default ecPolicy:  RS-DEFAULT-6-3-64k
>  Average block group size:  5.0
>  Missing block groups:  0
>  Corrupt block groups:  0
>  Missing internal blocks:   4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL. 
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is 
> unrecoverable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553330#comment-15553330
 ] 

Andrew Wang commented on HDFS-10971:


1, based on this code from S3AFileStatus:

{code}
  // Files
  public S3AFileStatus(long length, long modification_time, Path path,
  long blockSize) {
super(length, false, 1, blockSize, modification_time, path);
isEmptyDirectory = false;
  }
{code}

I think this is okay as long as we properly expose the durability level of EC 
files elsewhere. I've heard of operators for instance who want to verify there 
aren't any under-replicated files when doing maintenance operations.

> Distcp should not copy replication factor if source file is erasure coded
> -
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: distcp
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication factor field to 
> store erasure coding policy.
> Distcp copies the source file's replication factor to the destination if 
> {{-pr}} is specified. However, if the source file is EC, the replication 
> factor (which is EC policy) should not be replicated to the destination file. 
> When a HdfsFileStatus is converted to FileStatus, the replication factor is 
> set to 0 if it's an EC file.
> In fact, I will attach a test case that shows trying to replicate the 
> replication factor of an EC file results in an IOException: "Requested 
> replication factor of 0 is less than the required minimum of 1 for 
> /tmp/dst/dest2"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10975) fsck -list-corruptfileblocks does not report corrupt EC files

[Wei-Chiu Chuang (JIRA)]

Wei-Chiu Chuang created HDFS-10975:
--

 Summary: fsck -list-corruptfileblocks does not report corrupt EC 
files
 Key: HDFS-10975
 URL: https://issues.apache.org/jira/browse/HDFS-10975
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0-alpha2
Reporter: Wei-Chiu Chuang


HDFS-10826 fix fsck for corrupt EC files if no parameters are specified.

However, if I change the test case added in HDFS-10826 
(TestFsck#testFsckCorruptECFile) to run "fsck -list-corruptfileblocks", the 
same test test failed because fsck reports no corrupt files. 

Interestingly, if I run "fsck -files -blocks -replicaDetails" then the test 
passed and shows the corrupt file.

Need to fix the discrepancy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10955) Pass IIP for FSDirAttr methods

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553311#comment-15553311
 ] 

Hudson commented on HDFS-10955:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10561 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10561/])
HDFS-10955. Pass IIP for FSDirAttr methods. Contributed by Daryn Sharp. 
(kihwal: rev 48b9d5fd2a96728b1118be217ca597c4098e99ca)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java


> Pass IIP for FSDirAttr methods
> --
>
> Key: HDFS-10955
> URL: https://issues.apache.org/jira/browse/HDFS-10955
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10955.1.patch, HDFS-10955.patch
>
>
> Methods should always use the resolved IIP instead of re-solving the path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10912) Ozone:SCM: Add chill mode support to NodeManager.

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-10912:

Attachment: HDFS-10912-HDFS-7240.003.patch

[~xyao] Thank you for the comments and apologies for such late turnaround. I 
have addressed all issues mentioned by you in the last code review.


> Ozone:SCM: Add chill mode support to NodeManager.
> -
>
> Key: HDFS-10912
> URL: https://issues.apache.org/jira/browse/HDFS-10912
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Attachments: HDFS-10912-HDFS-7240.001.patch, 
> HDFS-10912-HDFS-7240.002.patch, HDFS-10912-HDFS-7240.003.patch
>
>
> Add chill mode support : That is to add the ability to force exit or enter 
> chill mode. As well as get the current chill mode status from node manager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10955) Pass IIP for FSDirAttr methods

[Kihwal Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated HDFS-10955:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
   2.8.0
   Status: Resolved  (was: Patch Available)

Committed to trunk, branch-2 and branch-2.8.

> Pass IIP for FSDirAttr methods
> --
>
> Key: HDFS-10955
> URL: https://issues.apache.org/jira/browse/HDFS-10955
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10955.1.patch, HDFS-10955.patch
>
>
> Methods should always use the resolved IIP instead of re-solving the path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10955) Pass IIP for FSDirAttr methods

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553241#comment-15553241
 ] 

Kihwal Lee commented on HDFS-10955:
---

+1

> Pass IIP for FSDirAttr methods
> --
>
> Key: HDFS-10955
> URL: https://issues.apache.org/jira/browse/HDFS-10955
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10955.1.patch, HDFS-10955.patch
>
>
> Methods should always use the resolved IIP instead of re-solving the path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded

[Zhe Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553245#comment-15553245
 ] 

Zhe Zhang commented on HDFS-10971:
--

Thanks for identifying the issue [~jojochuang] and [~andrew.wang]

The basic issue I see here is that EC files don't have a meaningful 
{{block_replication}}. Quick question, how is this field handled in S3 
filesystem?

> Distcp should not copy replication factor if source file is erasure coded
> -
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: distcp
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication factor field to 
> store erasure coding policy.
> Distcp copies the source file's replication factor to the destination if 
> {{-pr}} is specified. However, if the source file is EC, the replication 
> factor (which is EC policy) should not be replicated to the destination file. 
> When a HdfsFileStatus is converted to FileStatus, the replication factor is 
> set to 0 if it's an EC file.
> In fact, I will attach a test case that shows trying to replicate the 
> replication factor of an EC file results in an IOException: "Requested 
> replication factor of 0 is less than the required minimum of 1 for 
> /tmp/dst/dest2"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Issue Comment Deleted] (HDFS-10955) Pass IIP for FSDirAttr methods

[Kihwal Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated HDFS-10955:
--
Comment: was deleted

(was: +1)

> Pass IIP for FSDirAttr methods
> --
>
> Key: HDFS-10955
> URL: https://issues.apache.org/jira/browse/HDFS-10955
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10955.1.patch, HDFS-10955.patch
>
>
> Methods should always use the resolved IIP instead of re-solving the path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10955) Pass IIP for FSDirAttr methods

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553239#comment-15553239
 ] 

Kihwal Lee commented on HDFS-10955:
---

+1

> Pass IIP for FSDirAttr methods
> --
>
> Key: HDFS-10955
> URL: https://issues.apache.org/jira/browse/HDFS-10955
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10955.1.patch, HDFS-10955.patch
>
>
> Methods should always use the resolved IIP instead of re-solving the path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553234#comment-15553234
 ] 

Xiaobing Zhou commented on HDFS-10972:
--

v000 is posted for reviews, thanks.

> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10972
> URL: https://issues.apache.org/jira/browse/HDFS-10972
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10972.000.patch
>
>
> getDatanodeInfo should be tested in admin CLI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10972:
-
Status: Patch Available  (was: Open)

> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10972
> URL: https://issues.apache.org/jira/browse/HDFS-10972
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10972.000.patch
>
>
> getDatanodeInfo should be tested in admin CLI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553236#comment-15553236
 ] 

Wei-Chiu Chuang commented on HDFS-10971:


THat's a valid point. Currently HdfsFileStatus has a field to detect if a hdfs 
file is erasure coded. But FileStatus is a more high level abstraction for all 
types of file systems.

> Distcp should not copy replication factor if source file is erasure coded
> -
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: distcp
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication factor field to 
> store erasure coding policy.
> Distcp copies the source file's replication factor to the destination if 
> {{-pr}} is specified. However, if the source file is EC, the replication 
> factor (which is EC policy) should not be replicated to the destination file. 
> When a HdfsFileStatus is converted to FileStatus, the replication factor is 
> set to 0 if it's an EC file.
> In fact, I will attach a test case that shows trying to replicate the 
> replication factor of an EC file results in an IOException: "Requested 
> replication factor of 0 is less than the required minimum of 1 for 
> /tmp/dst/dest2"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10972:
-
Attachment: HDFS-10972.000.patch

> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10972
> URL: https://issues.apache.org/jira/browse/HDFS-10972
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10972.000.patch
>
>
> getDatanodeInfo should be tested in admin CLI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553164#comment-15553164
 ] 

Andrew Wang commented on HDFS-10971:


If the EC policy is showing up as the replication factor in FileStatus, that's 
a pretty incompatible change for users. Particularly since I don't see an 
"isErasureCoded" method on FileStatus similar to isEncrypted or getAclBit, so 
there's no way for the user to reasonably interpret this value without also 
querying the EC policy. This affects a lot more than just distcp.

We should consider squashing this and showing a dummy replication value like 
"1" instead for compatibility. Then, having an "isErasureCoded" method and 
perhaps a way of querying the policy. This is sort of like getAclBit, where 
there's getShort and toExtendedShort, with the former masking out the ACL bit.

[~zhz] / [~jingzhao] / [~drankye], thoughts welcomed.

> Distcp should not copy replication factor if source file is erasure coded
> -
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: distcp
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication factor field to 
> store erasure coding policy.
> Distcp copies the source file's replication factor to the destination if 
> {{-pr}} is specified. However, if the source file is EC, the replication 
> factor (which is EC policy) should not be replicated to the destination file. 
> When a HdfsFileStatus is converted to FileStatus, the replication factor is 
> set to 0 if it's an EC file.
> In fact, I will attach a test case that shows trying to replicate the 
> replication factor of an EC file results in an IOException: "Requested 
> replication factor of 0 is less than the required minimum of 1 for 
> /tmp/dst/dest2"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10955) Pass IIP for FSDirAttr methods

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553159#comment-15553159
 ] 

Hadoop QA commented on HDFS-10955:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 26s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 3 new + 273 unchanged - 10 fixed = 276 total (was 283) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m  0s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 44s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10955 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832019/HDFS-10955.1.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 7acc7f62e4c0 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 72a2ae6 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17049/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17049/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17049/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17049/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Pass IIP for FSDirAttr methods
> 

[jira] [Updated] (HDFS-10967) Add configuration for BlockPlacementPolicy to avoid near-full DataNodes

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-10967:
-
Attachment: HDFS-10967.poc.patch

Attaching a PoC patch just to demonstrate the idea.

> Add configuration for BlockPlacementPolicy to avoid near-full DataNodes
> ---
>
> Key: HDFS-10967
> URL: https://issues.apache.org/jira/browse/HDFS-10967
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>  Labels: balancer
> Attachments: HDFS-10967.poc.patch
>
>
> Large production clusters are likely to have heterogeneous nodes in terms of 
> storage capacity, memory, and CPU cores. It is not always possible to 
> proportionally ingest data into DataNodes based on their remaining storage 
> capacity. Therefore it's possible for a subset of DataNodes to be much closer 
> to full capacity than the rest.
> This heterogeneity is most likely rack-by-rack -- i.e. _m_ whole racks of 
> low-storage nodes and _n_ whole racks of high-storage nodes. So It'd be very 
> useful if we can lower the chance for those near-full DataNodes to become 
> destinations for the 2nd and 3rd replicas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10967) Add configuration for BlockPlacementPolicy to avoid near-full DataNodes

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-10967:
-
Status: Patch Available  (was: Open)

> Add configuration for BlockPlacementPolicy to avoid near-full DataNodes
> ---
>
> Key: HDFS-10967
> URL: https://issues.apache.org/jira/browse/HDFS-10967
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>  Labels: balancer
>
> Large production clusters are likely to have heterogeneous nodes in terms of 
> storage capacity, memory, and CPU cores. It is not always possible to 
> proportionally ingest data into DataNodes based on their remaining storage 
> capacity. Therefore it's possible for a subset of DataNodes to be much closer 
> to full capacity than the rest.
> This heterogeneity is most likely rack-by-rack -- i.e. _m_ whole racks of 
> low-storage nodes and _n_ whole racks of high-storage nodes. So It'd be very 
> useful if we can lower the chance for those near-full DataNodes to become 
> destinations for the 2nd and 3rd replicas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10939) Reduce performance penalty of encryption zones

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553120#comment-15553120
 ] 

Hudson commented on HDFS-10939:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10560 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10560/])
HDFS-10939. Reduce performance penalty of encryption zones. Contributed 
(kihwal: rev f32e9fc8f7150f0e889c0774b3ad712af26fbd65)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirEncryptionZoneOp.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EncryptionFaultInjector.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EncryptionZoneManager.java


> Reduce performance penalty of encryption zones
> --
>
> Key: HDFS-10939
> URL: https://issues.apache.org/jira/browse/HDFS-10939
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Affects Versions: 2.7
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10939.1.patch, HDFS-10939.2.patch, HDFS-10939.patch
>
>
> The encryption zone APIs should be optimized to extensively use IIPs to 
> eliminate path resolutions.  The performance penalties incurred by common 
> operations like creation of file statuses may be reduced by more extensive 
> short-circuiting of EZ lookups when no EZs exist.  All file creates should 
> not be subjected to the multi-stage locking performance penalty required only 
> for EDEK generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10939) Reduce performance penalty of encryption zones

[Kihwal Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated HDFS-10939:
--
Fix Version/s: 3.0.0-alpha2

> Reduce performance penalty of encryption zones
> --
>
> Key: HDFS-10939
> URL: https://issues.apache.org/jira/browse/HDFS-10939
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Affects Versions: 2.7
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10939.1.patch, HDFS-10939.2.patch, HDFS-10939.patch
>
>
> The encryption zone APIs should be optimized to extensively use IIPs to 
> eliminate path resolutions.  The performance penalties incurred by common 
> operations like creation of file statuses may be reduced by more extensive 
> short-circuiting of EZ lookups when no EZs exist.  All file creates should 
> not be subjected to the multi-stage locking performance penalty required only 
> for EDEK generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10939) Reduce performance penalty of encryption zones

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553090#comment-15553090
 ] 

Kihwal Lee commented on HDFS-10939:
---

Committed it to trunk, but branch-2 merge is not clean. The conflicts are 
mostly from EC-related context differences, but code moves make some of these 
beyond simple merge conflict resolution.  Please provide a branch-2 patch.

> Reduce performance penalty of encryption zones
> --
>
> Key: HDFS-10939
> URL: https://issues.apache.org/jira/browse/HDFS-10939
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Affects Versions: 2.7
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10939.1.patch, HDFS-10939.2.patch, HDFS-10939.patch
>
>
> The encryption zone APIs should be optimized to extensively use IIPs to 
> eliminate path resolutions.  The performance penalties incurred by common 
> operations like creation of file statuses may be reduced by more extensive 
> short-circuiting of EZ lookups when no EZs exist.  All file creates should 
> not be subjected to the multi-stage locking performance penalty required only 
> for EDEK generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10939) Reduce performance penalty of encryption zones

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553049#comment-15553049
 ] 

Kihwal Lee commented on HDFS-10939:
---

+1 the new patch looks good.

> Reduce performance penalty of encryption zones
> --
>
> Key: HDFS-10939
> URL: https://issues.apache.org/jira/browse/HDFS-10939
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Affects Versions: 2.7
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10939.1.patch, HDFS-10939.2.patch, HDFS-10939.patch
>
>
> The encryption zone APIs should be optimized to extensively use IIPs to 
> eliminate path resolutions.  The performance penalties incurred by common 
> operations like creation of file statuses may be reduced by more extensive 
> short-circuiting of EZ lookups when no EZs exist.  All file creates should 
> not be subjected to the multi-stage locking performance penalty required only 
> for EDEK generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10965) Add unit test for HDFS command 'dfsadmin -printTopology'

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552984#comment-15552984
 ] 

Hadoop QA commented on HDFS-10965:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated 
0 new + 167 unchanged - 34 fixed = 167 total (was 201) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m  2s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 54s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10965 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832004/HDFS-10965.003.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux a9b6cfacff31 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 72a2ae6 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17048/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17048/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17048/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add unit test for HDFS command 'dfsadmin -printTopology'
> 
>
> Key: HDFS-10965
> URL: https://issues.apache.org/jira/browse/HDFS-10965
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>

[jira] [Commented] (HDFS-10939) Reduce performance penalty of encryption zones

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552978#comment-15552978
 ] 

Hadoop QA commented on HDFS-10939:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 31s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 8 new + 407 unchanged - 9 fixed = 415 total (was 416) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 34s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 34s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tracing.TestTracing |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10939 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832003/HDFS-10939.2.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux f043c367c711 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 2d46c3f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17047/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17047/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17047/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17047/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Reduce performance penalty of encryption zones
> --
>
> Key: HDFS-10939
> URL: 

[jira] [Updated] (HDFS-10955) Pass IIP for FSDirAttr methods

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HDFS-10955:
---
Attachment: HDFS-10955.1.patch

Not sure how that happened.  Fixed.

> Pass IIP for FSDirAttr methods
> --
>
> Key: HDFS-10955
> URL: https://issues.apache.org/jira/browse/HDFS-10955
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10955.1.patch, HDFS-10955.patch
>
>
> Methods should always use the resolved IIP instead of re-solving the path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8707) Implement an async pure c++ HDFS client

[Bob Hansen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bob Hansen updated HDFS-8707:
-
Assignee: James Clampffer  (was: Bob Hansen)

> Implement an async pure c++ HDFS client
> ---
>
> Key: HDFS-8707
> URL: https://issues.apache.org/jira/browse/HDFS-8707
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: hdfs-client
>Reporter: Owen O'Malley
>Assignee: James Clampffer
>
> As part of working on the C++ ORC reader at ORC-3, we need an HDFS pure C++ 
> client that lets us do async io to HDFS. We want to start from the code that 
> Haohui's been working on at https://github.com/haohui/libhdfspp .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10974) Document replication factor for EC files.

[Wei-Chiu Chuang (JIRA)]

Wei-Chiu Chuang created HDFS-10974:
--

 Summary: Document replication factor for EC files.
 Key: HDFS-10974
 URL: https://issues.apache.org/jira/browse/HDFS-10974
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: documentation, erasure-coding
Affects Versions: 3.0.0-alpha1
Reporter: Wei-Chiu Chuang


The replication factor of an EC file is 0 from a user perspective. This affects 
all user visible commands and APIs, such as {{hdfs dfs -ls}}, {{hdfs dfs 
-stat}}, webhdfs, httpfs...etc.

In addition, FileSystem#setReplication API for an EC file is ignored. So is the 
command {{hdfs dfs -setrep}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552912#comment-15552912
 ] 

Mingliang Liu commented on HDFS-10903:
--

+1 pending on Jenkins.

> Replace config key literal strings with config key names II: hadoop hdfs
> 
>
> Key: HDFS-10903
> URL: https://issues.apache.org/jira/browse/HDFS-10903
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mingliang Liu
>Assignee: Chen Liang
>Priority: Minor
> Attachments: HADOOP-13644.001.patch, HDFS-10903.002.patch
>
>
> In *Hadoop HDFS*, there are some places that use config key literal strings 
> instead of config key names, e.g.
> {code:title=IOUtils.java}
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
> {code}
> We should replace places like this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552909#comment-15552909
 ] 

Mingliang Liu commented on HDFS-10903:
--

{{dfs.namenode.fslock.fair}} is being tracked in [HDFS-5239].

> Replace config key literal strings with config key names II: hadoop hdfs
> 
>
> Key: HDFS-10903
> URL: https://issues.apache.org/jira/browse/HDFS-10903
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mingliang Liu
>Assignee: Chen Liang
>Priority: Minor
> Attachments: HADOOP-13644.001.patch, HDFS-10903.002.patch
>
>
> In *Hadoop HDFS*, there are some places that use config key literal strings 
> instead of config key names, e.g.
> {code:title=IOUtils.java}
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
> {code}
> We should replace places like this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552903#comment-15552903
 ] 

Wei-Chiu Chuang commented on HDFS-10971:


The other place to check is RetriableFileCopyCommand#copyToFile. The 
destination file is created with source file replication factor if -pr is 
specified.

> Distcp should not copy replication factor if source file is erasure coded
> -
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: distcp
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication factor field to 
> store erasure coding policy.
> Distcp copies the source file's replication factor to the destination if 
> {{-pr}} is specified. However, if the source file is EC, the replication 
> factor (which is EC policy) should not be replicated to the destination file. 
> When a HdfsFileStatus is converted to FileStatus, the replication factor is 
> set to 0 if it's an EC file.
> In fact, I will attach a test case that shows trying to replicate the 
> replication factor of an EC file results in an IOException: "Requested 
> replication factor of 0 is less than the required minimum of 1 for 
> /tmp/dst/dest2"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen Liang updated HDFS-10903:
--
Attachment: HDFS-10903.002.patch

Uploaded v02 patch.

Whether to expose the key of {{dfs.namenode.fslock.fair}} is being tracked in, 
so excluded from this patch, also excluded deprecated keys such as 
{{mapred.temp.dir}}.

> Replace config key literal strings with config key names II: hadoop hdfs
> 
>
> Key: HDFS-10903
> URL: https://issues.apache.org/jira/browse/HDFS-10903
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mingliang Liu
>Assignee: Chen Liang
>Priority: Minor
> Attachments: HADOOP-13644.001.patch, HDFS-10903.002.patch
>
>
> In *Hadoop HDFS*, there are some places that use config key literal strings 
> instead of config key names, e.g.
> {code:title=IOUtils.java}
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
> {code}
> We should replace places like this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10972:
-
Description: getDatanodeInfo should be tested in admin CLI.

> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10972
> URL: https://issues.apache.org/jira/browse/HDFS-10972
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> getDatanodeInfo should be tested in admin CLI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-10971:
---
Description: 
The current erasure coding implementation uses replication refactor field to 
store erasure coding policy.

Distcp copies the source file's replication factor to the destination if 
{{-pr}} is specified. However, if the source file is EC, the replication factor 
(which is EC policy) should not be replicated to the destination file. When a 
HdfsFileStatus is converted to FileStatus, the replication factor is set to 0 
if it's an EC file.

In fact, I will attach a test case that shows trying to replicate the 
replication factor of an EC file results in an IOException: "Requested 
replication factor of 0 is less than the required minimum of 1 for 
/tmp/dst/dest2"

  was:
The current erasure coding implementation uses replication refactor field to 
store erasure coding policy.

Distcp copies the source file's replication refactor to the destination if -pr 
is specified. However, if the source file is EC, the replication factor (which 
is EC policy) should not be replicated to the destination file.

In fact, I will attach a test case that shows trying to replicate the 
replication factor of an EC file results in an IOException: "Requested 
replication factor of 0 is less than the required minimum of 1 for 
/tmp/dst/dest2"


> Distcp should not copy replication factor if source file is erasure coded
> -
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: distcp
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication refactor field to 
> store erasure coding policy.
> Distcp copies the source file's replication factor to the destination if 
> {{-pr}} is specified. However, if the source file is EC, the replication 
> factor (which is EC policy) should not be replicated to the destination file. 
> When a HdfsFileStatus is converted to FileStatus, the replication factor is 
> set to 0 if it's an EC file.
> In fact, I will attach a test case that shows trying to replicate the 
> replication factor of an EC file results in an IOException: "Requested 
> replication factor of 0 is less than the required minimum of 1 for 
> /tmp/dst/dest2"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10967) Add configuration for BlockPlacementPolicy to avoid near-full DataNodes

[Zhe Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552810#comment-15552810
 ] 

Zhe Zhang commented on HDFS-10967:
--

I'd like to discuss a few design perspectives. In general, I think this 
proposal is related to the existing {{considerLoad}} config knob, and we should 
leverage experience about how {{considerLoad}} has been used in production.
# Shall we add the logic only to 2nd and 3rd replicas? {{considerLoad}} applies 
to all replicas (including the first/local one). But I'm not sure if we should 
sacrifice locality if a DataNode is near-full.
# Which capacity metric should be used? Absolute amount of 
{{DatanodeInfo#remaining}} (indicating how many more replicas it can store) or 
the percentage of usage?
# Based on the selected metric, if a DataNode is near-full, should we simply 
avoid it, like how {{isGoodDatanode}} treats {{considerLoad}}? Or shall we just 
"toss the coin again" trying to find an emptier node?

> Add configuration for BlockPlacementPolicy to avoid near-full DataNodes
> ---
>
> Key: HDFS-10967
> URL: https://issues.apache.org/jira/browse/HDFS-10967
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>  Labels: balancer
>
> Large production clusters are likely to have heterogeneous nodes in terms of 
> storage capacity, memory, and CPU cores. It is not always possible to 
> proportionally ingest data into DataNodes based on their remaining storage 
> capacity. Therefore it's possible for a subset of DataNodes to be much closer 
> to full capacity than the rest.
> This heterogeneity is most likely rack-by-rack -- i.e. _m_ whole racks of 
> low-storage nodes and _n_ whole racks of high-storage nodes. So It'd be very 
> useful if we can lower the chance for those near-full DataNodes to become 
> destinations for the 2nd and 3rd replicas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-10971:
---
Description: 
The current erasure coding implementation uses replication factor field to 
store erasure coding policy.

Distcp copies the source file's replication factor to the destination if 
{{-pr}} is specified. However, if the source file is EC, the replication factor 
(which is EC policy) should not be replicated to the destination file. When a 
HdfsFileStatus is converted to FileStatus, the replication factor is set to 0 
if it's an EC file.

In fact, I will attach a test case that shows trying to replicate the 
replication factor of an EC file results in an IOException: "Requested 
replication factor of 0 is less than the required minimum of 1 for 
/tmp/dst/dest2"

  was:
The current erasure coding implementation uses replication refactor field to 
store erasure coding policy.

Distcp copies the source file's replication factor to the destination if 
{{-pr}} is specified. However, if the source file is EC, the replication factor 
(which is EC policy) should not be replicated to the destination file. When a 
HdfsFileStatus is converted to FileStatus, the replication factor is set to 0 
if it's an EC file.

In fact, I will attach a test case that shows trying to replicate the 
replication factor of an EC file results in an IOException: "Requested 
replication factor of 0 is less than the required minimum of 1 for 
/tmp/dst/dest2"


> Distcp should not copy replication factor if source file is erasure coded
> -
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: distcp
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication factor field to 
> store erasure coding policy.
> Distcp copies the source file's replication factor to the destination if 
> {{-pr}} is specified. However, if the source file is EC, the replication 
> factor (which is EC policy) should not be replicated to the destination file. 
> When a HdfsFileStatus is converted to FileStatus, the replication factor is 
> set to 0 if it's an EC file.
> In fact, I will attach a test case that shows trying to replicate the 
> replication factor of an EC file results in an IOException: "Requested 
> replication factor of 0 is less than the required minimum of 1 for 
> /tmp/dst/dest2"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10973) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

[Xiaobing Zhou (JIRA)]

Xiaobing Zhou created HDFS-10973:


 Summary: Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
 Key: HDFS-10973
 URL: https://issues.apache.org/jira/browse/HDFS-10973
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: fs, shell, test
Reporter: Xiaobing Zhou
Assignee: Xiaobing Zhou






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

[Xiaobing Zhou (JIRA)]

Xiaobing Zhou created HDFS-10972:


 Summary: Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
 Key: HDFS-10972
 URL: https://issues.apache.org/jira/browse/HDFS-10972
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: fs, shell, test
Reporter: Xiaobing Zhou
Assignee: Xiaobing Zhou






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10965) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10965:
-
Summary: Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'  (was: 
Add unit test for HDFS command 'dfsadmin -printTopology')

> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10965
> URL: https://issues.apache.org/jira/browse/HDFS-10965
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10965.000.patch, HDFS-10965.001.patch, 
> HDFS-10965.002.patch, HDFS-10965.003.patch
>
>
> DFSAdmin#printTopology should also be tested. This proposes adding it in 
> TestDFSAdmin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10965) Add unit test for HDFS command 'dfsadmin -printTopology'

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10965:
-
Summary: Add unit test for HDFS command 'dfsadmin -printTopology'  (was: 
Add unit test for HDFS command 'dfsadmin -getDatanodeInfo')

> Add unit test for HDFS command 'dfsadmin -printTopology'
> 
>
> Key: HDFS-10965
> URL: https://issues.apache.org/jira/browse/HDFS-10965
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10965.000.patch, HDFS-10965.001.patch, 
> HDFS-10965.002.patch, HDFS-10965.003.patch
>
>
> DFSAdmin#printTopology should also be tested. This proposes adding it in 
> TestDFSAdmin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-10971:
---
Labels: hdfs-ec-3.0-must-do  (was: )

> Distcp should not copy replication factor if source file is erasure coded
> -
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: distcp
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication refactor field to 
> store erasure coding policy.
> Distcp copies the source file's replication refactor to the destination if 
> -pr is specified. However, if the source file is EC, the replication factor 
> (which is EC policy) should not be replicated to the destination file.
> In fact, I will attach a test case that shows trying to replicate the 
> replication factor of an EC file results in an IOException: "Requested 
> replication factor of 0 is less than the required minimum of 1 for 
> /tmp/dst/dest2"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-10971:
---
Issue Type: Sub-task  (was: Bug)
Parent: HDFS-8031

> Distcp should not copy replication factor if source file is erasure coded
> -
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: distcp
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication refactor field to 
> store erasure coding policy.
> Distcp copies the source file's replication refactor to the destination if 
> -pr is specified. However, if the source file is EC, the replication factor 
> (which is EC policy) should not be replicated to the destination file.
> In fact, I will attach a test case that shows trying to replicate the 
> replication factor of an EC file results in an IOException: "Requested 
> replication factor of 0 is less than the required minimum of 1 for 
> /tmp/dst/dest2"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-10971:
---
Attachment: HDFS-10971.testcase.patch

Attach a unit test that demonstrates the failure. The test generates failure as 
follows:

{quote}
org.apache.hadoop.ipc.RemoteException(java.io.IOException): Requested 
replication factor of 0 is less than the required minimum of 1 for 
/tmp/dst/dest2
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.verifyReplication(BlockManager.java:1248)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setReplication(FSDirAttrOp.java:136)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setReplication(FSNamesystem.java:1986)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setReplication(NameNodeRpcServer.java:795)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setReplication(ClientNamenodeProtocolServerSideTranslatorPB.java:465)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:467)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:990)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1779)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2535)


at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481)
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1337)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy21.setReplication(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setReplication(ClientNamenodeProtocolTranslatorPB.java:355)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)
at com.sun.proxy.$Proxy22.setReplication(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.setReplication(DFSClient.java:1409)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$10.doCall(DistributedFileSystem.java:503)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$10.doCall(DistributedFileSystem.java:500)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.setReplication(DistributedFileSystem.java:510)
at 
org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:241)
at 
org.apache.hadoop.tools.util.TestDistCpUtils.testPreserveReplicationOnErasureCodedFile(TestDistCpUtils.java:577)
{quote}

> Distcp should not copy replication factor if source file is erasure coded
> -
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: distcp
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication refactor field to 
> store erasure coding policy.
> Distcp copies the source file's replication refactor to the destination if 
> -pr is specified. However, if the source file is EC, the replication factor 
> (which is EC policy) should not be replicated to the destination file.
> In fact, I will attach a test case that shows trying to replicate 

[jira] [Commented] (HDFS-10963) Reduce log level when network topology cannot find enough datanodes.

[Xiao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552765#comment-15552765
 ] 

Xiao Chen commented on HDFS-10963:
--

Thank you [~brahmareddy]!

> Reduce log level when network topology cannot find enough datanodes.
> 
>
> Key: HDFS-10963
> URL: https://issues.apache.org/jira/browse/HDFS-10963
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
>  Labels: supportability
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10963.01.patch
>
>
> Credit to [~qwertymaniac] who reported this.
> {quote}
> The change made by HDFS-10320 now causes all 1-rack cluster to keep printing 
> below:
>   WARNorg.apache.hadoop.net.NetworkTopology   
>   Failed to find datanode (scope="" excludedScope="/default").
> {quote}
> This was added in HDFS-10320 to replace an exception that caused NN 
> terminate. A warn log was added. But thinking closely, this should be debug 
> to reduce confusion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded

[Wei-Chiu Chuang (JIRA)]

Wei-Chiu Chuang created HDFS-10971:
--

 Summary: Distcp should not copy replication factor if source file 
is erasure coded
 Key: HDFS-10971
 URL: https://issues.apache.org/jira/browse/HDFS-10971
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: distcp
Affects Versions: 3.0.0-alpha1
Reporter: Wei-Chiu Chuang
Assignee: Wei-Chiu Chuang


The current erasure coding implementation uses replication refactor field to 
store erasure coding policy.

Distcp copies the source file's replication refactor to the destination if -pr 
is specified. However, if the source file is EC, the replication factor (which 
is EC policy) should not be replicated to the destination file.

In fact, I will attach a test case that shows trying to replicate the 
replication factor of an EC file results in an IOException: "Requested 
replication factor of 0 is less than the required minimum of 1 for 
/tmp/dst/dest2"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9878) Data transfer encryption with AES 192: Invalid key length.

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-9878:

Assignee: Harsh J

> Data transfer encryption with AES 192: Invalid key length.
> --
>
> Key: HDFS-9878
> URL: https://issues.apache.org/jira/browse/HDFS-9878
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode, hdfs-client
>Affects Versions: 2.7.2
> Environment: OS: Ubuntu 14.04
> /hadoop-2.7.2/bin$ uname -a
> Linux wkstn-kpalaniappan 3.13.0-79-generic #123-Ubuntu SMP Fri Feb 19 
> 14:27:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> /hadoop-2.7.2/bin$ java -version
> java version "1.7.0_95"
> OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-0ubuntu0.14.04.1)
> OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
> Hadoop version: 2.7.2
>Reporter: Karthik Palaniappan
>Assignee: Harsh J
>
> Configuring aes 128 or aes 256 encryption 
> (dfs.encrypt.data.transfer.cipher.key.bitlength = [128, 256]) works perfectly 
> fine. Trying to use AES 192 generates this exception on the datanode:
> 16/02/29 17:34:10 ERROR datanode.DataNode: 
> wkstn-kpalaniappan:50010:DataXceiver error processing unknown operation  src: 
> /127.0.0.1:57237 dst: /127.0.0.1:50010
> java.lang.IllegalArgumentException: Invalid key length.
>   at org.apache.hadoop.crypto.OpensslCipher.init(Native Method)
>   at org.apache.hadoop.crypto.OpensslCipher.init(OpensslCipher.java:176)
>   at 
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec$OpensslAesCtrCipher.init(OpensslAesCtrCryptoCodec.java:116)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.updateDecryptor(CryptoInputStream.java:290)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.resetStreamOffset(CryptoInputStream.java:303)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:128)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:109)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:133)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.createStreamPair(DataTransferSaslUtil.java:345)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:396)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:110)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:193)
>   at java.lang.Thread.run(Thread.java:745)
> And this exception on the client:
> /hadoop-2.7.2/bin$ ./hdfs dfs -copyFromLocal ~/.vimrc /vimrc
> 16/02/29 17:34:10 WARN hdfs.DFSClient: DataStreamer Exception
> java.lang.IllegalArgumentException: Invalid key length.
>   at org.apache.hadoop.crypto.OpensslCipher.init(Native Method)
>   at org.apache.hadoop.crypto.OpensslCipher.init(OpensslCipher.java:176)
>   at 
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec$OpensslAesCtrCipher.init(OpensslAesCtrCryptoCodec.java:116)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.updateDecryptor(CryptoInputStream.java:290)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.resetStreamOffset(CryptoInputStream.java:303)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:128)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:109)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:133)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.createStreamPair(DataTransferSaslUtil.java:345)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:490)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1266)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
> 

[jira] [Updated] (HDFS-10965) Add unit test for HDFS command 'dfsadmin -printTopology'

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10965:
-
Attachment: HDFS-10965.003.patch

Posted patch v003, using ToolRunner to invoke it instead of calling it directly.

> Add unit test for HDFS command 'dfsadmin -printTopology'
> 
>
> Key: HDFS-10965
> URL: https://issues.apache.org/jira/browse/HDFS-10965
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10965.000.patch, HDFS-10965.001.patch, 
> HDFS-10965.002.patch, HDFS-10965.003.patch
>
>
> DFSAdmin#printTopology should also be tested. This proposes adding it in 
> TestDFSAdmin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10939) Reduce performance penalty of encryption zones

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HDFS-10939:
---
Attachment: HDFS-10939.2.patch

Address kihwal's comments.  Had to add another fault injector method to keep 
the implementation details encapsulated in a single method.

> Reduce performance penalty of encryption zones
> --
>
> Key: HDFS-10939
> URL: https://issues.apache.org/jira/browse/HDFS-10939
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Affects Versions: 2.7
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10939.1.patch, HDFS-10939.2.patch, HDFS-10939.patch
>
>
> The encryption zone APIs should be optimized to extensively use IIPs to 
> eliminate path resolutions.  The performance penalties incurred by common 
> operations like creation of file statuses may be reduced by more extensive 
> short-circuiting of EZ lookups when no EZs exist.  All file creates should 
> not be subjected to the multi-stage locking performance penalty required only 
> for EDEK generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode

[Erik Krogen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552636#comment-15552636
 ] 

Erik Krogen edited comment on HDFS-10921 at 10/6/16 5:50 PM:
-

[~shahrs87], I tried that previously, but when running the entire Test file in 
one run with that modification, the first two tests would pass and subsequent 
ones would fail... I was thinking it may have something to do with calling 
{{setImageLoaded}} causing some sort of inconsistent state in the Namesystem? 
The error I get is:
{code}
java.lang.IllegalStateException: Cannot skip to less than the current value 
(=1073741826), where newValue=1073741825
at 
org.apache.hadoop.util.SequentialNumber.skipTo(SequentialNumber.java:58)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockIdManager.setLastAllocatedContiguousBlockId(BlockIdManager.java:112)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:806)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:240)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:149)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:838)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:695)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:291)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1012)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:666)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:650)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:712)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:928)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:907)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1624)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2073)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNodes(MiniDFSCluster.java:2028)
at 
org.apache.hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate.testTruncateOverQuota(TestDiskspaceQuotaUpdate.java:347)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
So it seems to me that it creates an inconsistency between the edit log and the 
image?


was (Author: xkrogen):
[~shahrs87], I tried that previously, but when running the entire Test file in 
one run with that modification, the first two tests would pass and subsequent 
ones would fail... I was thinking it may have something to do with calling 
{{setImageLoaded}} causing some sort of inconsistent state in the Namesystem?

> TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
> 
>
> Key: HDFS-10921
> URL: https://issues.apache.org/jira/browse/HDFS-10921
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch
>
>
> Test fails intermittently because the NN is still in safe mode. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode

[Erik Krogen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552636#comment-15552636
 ] 

Erik Krogen commented on HDFS-10921:


[~shahrs87], I tried that previously, but when running the entire Test file in 
one run with that modification, the first two tests would pass and subsequent 
ones would fail... I was thinking it may have something to do with calling 
{{setImageLoaded}} causing some sort of inconsistent state in the Namesystem?

> TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
> 
>
> Key: HDFS-10921
> URL: https://issues.apache.org/jira/browse/HDFS-10921
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch
>
>
> Test fails intermittently because the NN is still in safe mode. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-10967) Add configuration for BlockPlacementPolicy to avoid near-full DataNodes

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang reassigned HDFS-10967:


Assignee: Zhe Zhang

> Add configuration for BlockPlacementPolicy to avoid near-full DataNodes
> ---
>
> Key: HDFS-10967
> URL: https://issues.apache.org/jira/browse/HDFS-10967
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>  Labels: balancer
>
> Large production clusters are likely to have heterogeneous nodes in terms of 
> storage capacity, memory, and CPU cores. It is not always possible to 
> proportionally ingest data into DataNodes based on their remaining storage 
> capacity. Therefore it's possible for a subset of DataNodes to be much closer 
> to full capacity than the rest.
> This heterogeneity is most likely rack-by-rack -- i.e. _m_ whole racks of 
> low-storage nodes and _n_ whole racks of high-storage nodes. So It'd be very 
> useful if we can lower the chance for those near-full DataNodes to become 
> destinations for the 2nd and 3rd replicas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10970) Update jackson from 1.9.13 to 2.x in hadoop-hdfs

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552575#comment-15552575
 ] 

Hadoop QA commented on HDFS-10970:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
25s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 164 
unchanged - 1 fixed = 164 total (was 165) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
54s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 59m 
43s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
44s{color} | {color:green} hadoop-hdfs-nfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m  7s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10970 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831606/HADOOP-13683.01.patch 
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  findbugs  checkstyle  |
| uname | Linux fadeee598a96 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 2cc841f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17046/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client 
hadoop-hdfs-project/hadoop-hdfs 

[jira] [Updated] (HDFS-10745) Directly resolve paths into INodesInPath

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-10745:
-
   Resolution: Fixed
Fix Version/s: 2.7.4
   Status: Resolved  (was: Patch Available)

Addressed all comments from [~shv], also verified all reported test failures, 
and added entry to CHANGES.txt.

I just committed the v6 branch-2.7 patch. Thanks [~shv] for the review!

> Directly resolve paths into INodesInPath
> 
>
> Key: HDFS-10745
> URL: https://issues.apache.org/jira/browse/HDFS-10745
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1
>
> Attachments: HDFS-10745-branch-2.7.01.patch, 
> HDFS-10745-branch-2.7.02.patch, HDFS-10745-branch-2.7.03.patch, 
> HDFS-10745-branch-2.7.04.patch, HDFS-10745-branch-2.7.05.patch, 
> HDFS-10745-branch-2.7.06.patch, HDFS-10745-branch-2.7.patch, 
> HDFS-10745.2.patch, HDFS-10745.branch-2.patch, HDFS-10745.patch
>
>
> The intermediate resolution to a string, only to be decomposed by 
> {{INodesInPath}} back into a byte[][] can be eliminated by resolving directly 
> to an IIP.  The IIP will contain the resolved path if required.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10967) Add configuration for BlockPlacementPolicy to avoid near-full DataNodes

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-10967:
-
Description: 
Large production clusters are likely to have heterogeneous nodes in terms of 
storage capacity, memory, and CPU cores. It is not always possible to 
proportionally ingest data into DataNodes based on their remaining storage 
capacity. Therefore it's possible for a subset of DataNodes to be much closer 
to full capacity than the rest.

This heterogeneity is most likely rack-by-rack -- i.e. _m_ whole racks of 
low-storage nodes and _n_ whole racks of high-storage nodes. So It'd be very 
useful if we can lower the chance for those near-full DataNodes to become 
destinations for the 2nd and 3rd replicas.

  was:
Large production clusters are likely to have heterogeneous nodes in terms of 
storage capacity, memory, and CPU cores. It is not always possible to 
proportionally ingest data into DataNodes based on their remaining storage 
capacity. Therefore it's possible for a subset of DataNodes to be much closer 
to full capacity than the rest.

This heterogeneity is most likely rack-by-rack -- i.e. _m_ whole racks with 
low-storage nodes and _n_ whole racks with high-storage nodes. So It'd be very 
useful if we can lower the chance for those near-full DataNodes to become 
destinations for the 2nd and 3rd replicas.


> Add configuration for BlockPlacementPolicy to avoid near-full DataNodes
> ---
>
> Key: HDFS-10967
> URL: https://issues.apache.org/jira/browse/HDFS-10967
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Zhe Zhang
>  Labels: balancer
>
> Large production clusters are likely to have heterogeneous nodes in terms of 
> storage capacity, memory, and CPU cores. It is not always possible to 
> proportionally ingest data into DataNodes based on their remaining storage 
> capacity. Therefore it's possible for a subset of DataNodes to be much closer 
> to full capacity than the rest.
> This heterogeneity is most likely rack-by-rack -- i.e. _m_ whole racks of 
> low-storage nodes and _n_ whole racks of high-storage nodes. So It'd be very 
> useful if we can lower the chance for those near-full DataNodes to become 
> destinations for the 2nd and 3rd replicas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10967) Add configuration for BlockPlacementPolicy to avoid near-full DataNodes

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-10967:
-
Description: 
Large production clusters are likely to have heterogeneous nodes in terms of 
storage capacity, memory, and CPU cores. It is not always possible to 
proportionally ingest data into DataNodes based on their remaining storage 
capacity. Therefore it's possible for a subset of DataNodes to be much closer 
to full capacity than the rest.

This heterogeneity is most likely rack-by-rack -- i.e. _m_ whole racks with 
low-storage nodes and _n_ whole racks with high-storage nodes. So It'd be very 
useful if we can lower the chance for those near-full DataNodes to become 
destinations for the 2nd and 3rd replicas.

  was:
Large production clusters are likely to have heterogeneous nodes in terms of 
storage capacity, memory, and CPU cores. It is not always possible to 
proportionally ingest data into DataNodes based on their remaining storage 
capacity. Therefore it's possible for a subset of DataNodes to be much closer 
to full capacity than the rest.

Notice that this heterogeneity is most likely rack-by-rack -- i.e. _m_ whole 
racks with low-storage nodes and _n_ whole racks with high-storage nodes. So 
It'd be very useful if we can lower the chance for those near-full DataNodes to 
become destinations for the 2nd and 3rd replicas.


> Add configuration for BlockPlacementPolicy to avoid near-full DataNodes
> ---
>
> Key: HDFS-10967
> URL: https://issues.apache.org/jira/browse/HDFS-10967
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Zhe Zhang
>  Labels: balancer
>
> Large production clusters are likely to have heterogeneous nodes in terms of 
> storage capacity, memory, and CPU cores. It is not always possible to 
> proportionally ingest data into DataNodes based on their remaining storage 
> capacity. Therefore it's possible for a subset of DataNodes to be much closer 
> to full capacity than the rest.
> This heterogeneity is most likely rack-by-rack -- i.e. _m_ whole racks with 
> low-storage nodes and _n_ whole racks with high-storage nodes. So It'd be 
> very useful if we can lower the chance for those near-full DataNodes to 
> become destinations for the 2nd and 3rd replicas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode

[Rushabh S Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552396#comment-15552396
 ] 

Rushabh S Shah commented on HDFS-10921:
---

bq. So the only problem that we have is the issue that Rushabh S Shah raised 
about Namesystem pollution by bad tests.
I looked at the code.
{code:title=FSNamesystem.java|borderStyle=solid}
/**
   * Clear all loaded data
   */
  void clear() {
dir.reset();
dtSecretManager.reset();
leaseManager.removeAllLeases();
snapshotManager.clearSnapshottableDirs();
cacheManager.clear();
ecPolicyManager.clear();
setImageLoaded(false);
blockManager.clear();
  }
{code}
bq. setImageLoaded(false);
This line from {{clear()}} method is the problem.

{code:title=FSDirectory.java|borderStyle=solid}
 void updateCount(INodesInPath iip, int numOfINodes,
QuotaCounts counts, boolean checkQuota)
throws QuotaExceededException {
assert hasWriteLock();
if (!namesystem.isImageLoaded()) {
  //still initializing. do not check or update quotas.
  return;
}
if (numOfINodes > iip.length()) {
  numOfINodes = iip.length();
}
if (checkQuota && !skipQuotaCheck) {
  verifyQuota(iip, numOfINodes, counts, null);
}
unprotectedUpdateCount(iip, numOfINodes, counts);
  }
{code}

{quote}
 if (!namesystem.isImageLoaded()) \{
  //still initializing. do not check or update quotas.
  return;
\}
{quote}
While creating a directory or file, it tries to update the usageCount.
But since in the {{FSNamesystem#clear()}}, we set the imageLoaded flag to 
false, the {{FSDirectory#updateCount}} will return and 
{{FSDirectory#unprotectedUpdateCount}} wont be called and the usage count will 
not be updated.

One thing we can do is:
After calling FSNamesystem#clear in the @Before block, we can call 
{{cluster.getNamesystem().setImageLoaded(true);}}
I just tried running one test case 
{{TestDiskspaceQuotaUpdate#testQuotaIssuesWhileCommitting}} and it passed.
Hope this helps.


> TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
> 
>
> Key: HDFS-10921
> URL: https://issues.apache.org/jira/browse/HDFS-10921
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch
>
>
> Test fails intermittently because the NN is still in safe mode. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10893) Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HDFS-10893:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.9.0
   Status: Resolved  (was: Patch Available)

Committed to {{trunk}} and {{branch-2}}. Thanks for review, [~jnp].

> Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test
> 
>
> Key: HDFS-10893
> URL: https://issues.apache.org/jira/browse/HDFS-10893
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.9.0
>
> Attachments: HDFS-10893-branch-2.000.patch, 
> HDFS-10893-branch-2.001.patch, HDFS-10893-branch-2.002.patch, 
> HDFS-10893-branch-2.002.patch, HDFS-10893.000.patch, HDFS-10893.001.patch, 
> HDFS-10893.002.patch
>
>
> It seems that setting up MiniDFSCluser once for all commands test will reduce 
> the total time. To share a global cluster, the tests should use individual 
> test directories to avoid conflict between test cases. Meanwhile, the 
> MiniDFSCluster should not use the default root data directory; or else tests 
> are not able to launch another cluster(s) by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Moved] (HDFS-10970) Update jackson from 1.9.13 to 2.x in hadoop-hdfs

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka moved HADOOP-13683 to HDFS-10970:
---

Target Version/s: 3.0.0-alpha2  (was: 3.0.0-alpha2)
 Component/s: (was: build)
  build
 Key: HDFS-10970  (was: HADOOP-13683)
 Project: Hadoop HDFS  (was: Hadoop Common)

> Update jackson from 1.9.13 to 2.x in hadoop-hdfs
> 
>
> Key: HDFS-10970
> URL: https://issues.apache.org/jira/browse/HDFS-10970
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: build
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
> Attachments: HADOOP-13683.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9820) Improve distcp to support efficient restore to an earlier snapshot

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552241#comment-15552241
 ] 

Yongjun Zhang commented on HDFS-9820:
-

Hi [~andrew.wang],

As I verbally addressed your question
[quote}
Is there a situation where we'd want to pass two different clusters for "src" 
and "tgt" to rdiff? They need to both have identical "s1" base states anyway.
{quote}
The reason is that a user reported copying from a different cluster is faster,  
so I hope we can support the flexibility to copy from either the same cluster's 
snapshot or a different cluster's snapshot. It's common for user to have a 
mirroring cluster, one is production, one is a backup.

I will address most of the checkstyle issue in next rev, but one thing I 
followed the existing test is to use "_" in file name variables to indicate the 
hierarchy, which seems easier to read, so I'd like to keep that.

Thanks.





> Improve distcp to support efficient restore to an earlier snapshot
> --
>
> Key: HDFS-9820
> URL: https://issues.apache.org/jira/browse/HDFS-9820
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: distcp
>Affects Versions: 2.6.4
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-9820.001.patch, HDFS-9820.002.patch, 
> HDFS-9820.003.patch, HDFS-9820.004.patch, HDFS-9820.005.patch, 
> HDFS-9820.006.patch
>
>
> A common use scenario (scenaio 1): 
> # create snapshot sx in clusterX, 
> # do some experiemnts in clusterX, which creates some files. 
> # throw away the files changed and go back to sx.
> Another scenario (scenario 2) is, there is a production cluster and a backup 
> cluster, we periodically sync up the data from production cluster to the 
> backup cluster with distcp. 
> The cluster in scenario 1 could be the backup cluster in scenario 2.
> For scenario 1:
> HDFS-4167 intends to restore HDFS to the most recent snapshot, and there are 
> some complexity and challenges.  Before that jira is implemented, we count on 
> distcp to copy from snapshot to the current state. However, the performance 
> of this operation could be very bad because we have to go through all files 
> even if we only changed a few files.
> For scenario 2:
> HDFS-7535 improved distcp performance by avoiding copying files that changed 
> name since last backup.
> On top of HDFS-7535, HDFS-8828 improved distcp performance when copying data 
> from source to target cluster, by only copying changed files since last 
> backup. The way it works is use snapshot diff to find out all files changed, 
> and copy the changed files only.
> See 
> https://blog.cloudera.com/blog/2015/12/distcp-performance-improvements-in-apache-hadoop/
> This jira is to propose a variation of HDFS-8828, to find out the files 
> changed in target cluster since last snapshot sx, and copy these from 
> snapshot sx of either the source or the target cluster, to restore target 
> cluster's current state to sx. 
> Specifically,
> If a file/dir is
> - renamed, rename it back
> - created in target cluster, delete it
> - modified, put it to the copy list
> - run distcp with the copy list, copy from the source cluster's corresponding 
> snapshot
> This could be a new command line switch -rdiff in distcp.
> As a native restore feature, HDFS-4167 would still be ideal to have. However, 
>  HDFS-9820 would hopefully be easier to implement, before HDFS-4167 is in 
> place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-9820) Improve distcp to support efficient restore to an earlier snapshot

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552241#comment-15552241
 ] 

Yongjun Zhang edited comment on HDFS-9820 at 10/6/16 3:35 PM:
--

Hi [~andrew.wang],

As I verbally addressed your question
{quote}
Is there a situation where we'd want to pass two different clusters for "src" 
and "tgt" to rdiff? They need to both have identical "s1" base states anyway.
{quote}

The reason is that a user reported copying from a different cluster is faster,  
so I hope we can support the flexibility to copy from either the same cluster's 
snapshot or a different cluster's snapshot. It's common for user to have a 
mirroring cluster, one is production, one is a backup.

I will address most of the checkstyle issue in next rev, but one thing I 
followed the existing test is to use "_" in file name variables to indicate the 
hierarchy, which seems easier to read, so I'd like to keep that.

Thanks.






was (Author: yzhangal):
Hi [~andrew.wang],

As I verbally addressed your question
[quote}
Is there a situation where we'd want to pass two different clusters for "src" 
and "tgt" to rdiff? They need to both have identical "s1" base states anyway.
{quote}
The reason is that a user reported copying from a different cluster is faster,  
so I hope we can support the flexibility to copy from either the same cluster's 
snapshot or a different cluster's snapshot. It's common for user to have a 
mirroring cluster, one is production, one is a backup.

I will address most of the checkstyle issue in next rev, but one thing I 
followed the existing test is to use "_" in file name variables to indicate the 
hierarchy, which seems easier to read, so I'd like to keep that.

Thanks.





> Improve distcp to support efficient restore to an earlier snapshot
> --
>
> Key: HDFS-9820
> URL: https://issues.apache.org/jira/browse/HDFS-9820
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: distcp
>Affects Versions: 2.6.4
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-9820.001.patch, HDFS-9820.002.patch, 
> HDFS-9820.003.patch, HDFS-9820.004.patch, HDFS-9820.005.patch, 
> HDFS-9820.006.patch
>
>
> A common use scenario (scenaio 1): 
> # create snapshot sx in clusterX, 
> # do some experiemnts in clusterX, which creates some files. 
> # throw away the files changed and go back to sx.
> Another scenario (scenario 2) is, there is a production cluster and a backup 
> cluster, we periodically sync up the data from production cluster to the 
> backup cluster with distcp. 
> The cluster in scenario 1 could be the backup cluster in scenario 2.
> For scenario 1:
> HDFS-4167 intends to restore HDFS to the most recent snapshot, and there are 
> some complexity and challenges.  Before that jira is implemented, we count on 
> distcp to copy from snapshot to the current state. However, the performance 
> of this operation could be very bad because we have to go through all files 
> even if we only changed a few files.
> For scenario 2:
> HDFS-7535 improved distcp performance by avoiding copying files that changed 
> name since last backup.
> On top of HDFS-7535, HDFS-8828 improved distcp performance when copying data 
> from source to target cluster, by only copying changed files since last 
> backup. The way it works is use snapshot diff to find out all files changed, 
> and copy the changed files only.
> See 
> https://blog.cloudera.com/blog/2015/12/distcp-performance-improvements-in-apache-hadoop/
> This jira is to propose a variation of HDFS-8828, to find out the files 
> changed in target cluster since last snapshot sx, and copy these from 
> snapshot sx of either the source or the target cluster, to restore target 
> cluster's current state to sx. 
> Specifically,
> If a file/dir is
> - renamed, rename it back
> - created in target cluster, delete it
> - modified, put it to the copy list
> - run distcp with the copy list, copy from the source cluster's corresponding 
> snapshot
> This could be a new command line switch -rdiff in distcp.
> As a native restore feature, HDFS-4167 would still be ideal to have. However, 
>  HDFS-9820 would hopefully be easier to implement, before HDFS-4167 is in 
> place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10922) Adding additional unit tests for Trash (II)

[Weiwei Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552077#comment-15552077
 ] 

Weiwei Yang commented on HDFS-10922:


Sounds good to me, will work on that. I was adding the permission tests for #1 
in hdfs, because I was not sure how to test permission in localFS, any hints on 
that? I will also take a look shortly. Thanks for the suggestions.

> Adding additional unit tests for Trash (II)
> ---
>
> Key: HDFS-10922
> URL: https://issues.apache.org/jira/browse/HDFS-10922
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Xiaoyu Yao
>Assignee: Weiwei Yang
> Attachments: HDFS-10922.02.patch, HDFS-10922.03.patch, 
> HDFS-10922.04.patch, HDFS-10922.05.patch, HDFS-10922.06.patch
>
>
> This ticket is opened to track adding unit tests for Trash.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10957) Retire BKJM from trunk

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552052#comment-15552052
 ] 

Hudson commented on HDFS-10957:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10555 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10555/])
HDFS-10957. Retire BKJM from trunk (Vinayakumar B) (vinayakumarb: rev 
311954883f714973784432589896553eb320b597)
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/CurrentInprogress.java
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/MaxTxId.java
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestCurrentInprogress.java
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/EditLogLedgerMetadata.java
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBootstrapStandbyWithBKJM.java
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperSpeculativeRead.java
* (edit) hadoop-project/pom.xml
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* (edit) hadoop-hdfs-project/pom.xml
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/dev-support/findbugsExcludeFile.xml
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSHighAvailabilityWithNFS.md
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java
* (delete) hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperConfiguration.java
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/proto/bkjournal.proto
* (delete) 
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperEditLogStreams.java
* (delete) hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt


> Retire BKJM from trunk
> --
>
> Key: HDFS-10957
> URL: https://issues.apache.org/jira/browse/HDFS-10957
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-10957-01.patch, HDFS-10957-02.patch
>
>
> Based on discussions done in mailing list 
> [here|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3c1288296033.5942327.1469727453010.javamail.ya...@mail.yahoo.com%3E]
>  BKJM is no longer required as QJM supports the required functionality with 
> minimum deployment overhead.
> This Jira is for tracking purpose.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10939) Reduce performance penalty of encryption zones

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552022#comment-15552022
 ] 

Daryn Sharp commented on HDFS-10939:


Agreed, I wasn't very fond of the locking asymmetry.  Original implementation 
was similar to your suggestion of relock in the finally but didn't like 
relocking/unlocking on failure. You're right that errors should be a rare case. 
 Will update patch.

> Reduce performance penalty of encryption zones
> --
>
> Key: HDFS-10939
> URL: https://issues.apache.org/jira/browse/HDFS-10939
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Affects Versions: 2.7
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10939.1.patch, HDFS-10939.patch
>
>
> The encryption zone APIs should be optimized to extensively use IIPs to 
> eliminate path resolutions.  The performance penalties incurred by common 
> operations like creation of file statuses may be reduced by more extensive 
> short-circuiting of EZ lookups when no EZs exist.  All file creates should 
> not be subjected to the multi-stage locking performance penalty required only 
> for EDEK generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10957) Retire BKJM from trunk

[Vinayakumar B (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552017#comment-15552017
 ] 

Vinayakumar B commented on HDFS-10957:
--

Committed to trunk.
Thanks [~brahmareddy] and [~kihwal]
And everyone replied in mailing thread.

> Retire BKJM from trunk
> --
>
> Key: HDFS-10957
> URL: https://issues.apache.org/jira/browse/HDFS-10957
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-10957-01.patch, HDFS-10957-02.patch
>
>
> Based on discussions done in mailing list 
> [here|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3c1288296033.5942327.1469727453010.javamail.ya...@mail.yahoo.com%3E]
>  BKJM is no longer required as QJM supports the required functionality with 
> minimum deployment overhead.
> This Jira is for tracking purpose.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10957) Retire BKJM from trunk

[Brahma Reddy Battula (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551954#comment-15551954
 ] 

Brahma Reddy Battula commented on HDFS-10957:
-

Latest patch looks good to me , +1..Thanks [~vinayrpet] 

> Retire BKJM from trunk
> --
>
> Key: HDFS-10957
> URL: https://issues.apache.org/jira/browse/HDFS-10957
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-10957-01.patch, HDFS-10957-02.patch
>
>
> Based on discussions done in mailing list 
> [here|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3c1288296033.5942327.1469727453010.javamail.ya...@mail.yahoo.com%3E]
>  BKJM is no longer required as QJM supports the required functionality with 
> minimum deployment overhead.
> This Jira is for tracking purpose.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10963) Reduce log level when network topology cannot find enough datanodes.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551839#comment-15551839
 ] 

Hudson commented on HDFS-10963:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10553 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10553/])
HDFS-10963. Reduce log level when network topology cannot find enough (brahma: 
rev b90fc70d671481564e468550c770c925f25d7db0)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java


> Reduce log level when network topology cannot find enough datanodes.
> 
>
> Key: HDFS-10963
> URL: https://issues.apache.org/jira/browse/HDFS-10963
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
>  Labels: supportability
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10963.01.patch
>
>
> Credit to [~qwertymaniac] who reported this.
> {quote}
> The change made by HDFS-10320 now causes all 1-rack cluster to keep printing 
> below:
>   WARNorg.apache.hadoop.net.NetworkTopology   
>   Failed to find datanode (scope="" excludedScope="/default").
> {quote}
> This was added in HDFS-10320 to replace an exception that caused NN 
> terminate. A warn log was added. But thinking closely, this should be debug 
> to reduce confusion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10963) Reduce log level when network topology cannot find enough datanodes.

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HDFS-10963:

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
   2.8.0
   Status: Resolved  (was: Patch Available)

Committed to trunk,branch-2 and branch-2.8. [~xiaochen] thanks for your 
contribution.

> Reduce log level when network topology cannot find enough datanodes.
> 
>
> Key: HDFS-10963
> URL: https://issues.apache.org/jira/browse/HDFS-10963
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
>  Labels: supportability
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10963.01.patch
>
>
> Credit to [~qwertymaniac] who reported this.
> {quote}
> The change made by HDFS-10320 now causes all 1-rack cluster to keep printing 
> below:
>   WARNorg.apache.hadoop.net.NetworkTopology   
>   Failed to find datanode (scope="" excludedScope="/default").
> {quote}
> This was added in HDFS-10320 to replace an exception that caused NN 
> terminate. A warn log was added. But thinking closely, this should be debug 
> to reduce confusion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9337) Should check required params in WebHDFS to avoid NPE

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551781#comment-15551781
 ] 

Hadoop QA commented on HDFS-9337:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 38s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 17s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHDFSXAttr |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-9337 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831937/HDFS-9337_06.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 87fabe0ff45b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 272a217 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17045/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17045/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17045/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Should check required params in WebHDFS to avoid NPE
> 
>
> Key: HDFS-9337
> URL: https://issues.apache.org/jira/browse/HDFS-9337
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Jagadesh 

[jira] [Commented] (HDFS-10933) Refactor TestFsck

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551747#comment-15551747
 ] 

Hadoop QA commented on HDFS-10933:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
24s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated 
0 new + 0 unchanged - 277 fixed = 0 total (was 277) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 63m  
6s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 51s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10933 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831934/HDFS-10933.2.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 1b74569264f5 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 272a217 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17044/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17044/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Refactor TestFsck
> -
>
> Key: HDFS-10933
> URL: https://issues.apache.org/jira/browse/HDFS-10933
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Attachments: HDFS-10933.1.patch, HDFS-10933.2.patch, 
> HDFS-10933.WIP.1.patch
>
>
> {{TestFsck}} should be refactored.
> - use @Before @After annotations
> - improve loggings
> - fix checkstyle warnings
> etc.



--
This 

[jira] [Commented] (HDFS-9878) Data transfer encryption with AES 192: Invalid key length.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551661#comment-15551661
 ] 

ASF GitHub Bot commented on HDFS-9878:
--

GitHub user QwertyManiac opened a pull request:

https://github.com/apache/hadoop/pull/135

HDFS-9878. Add support for AES-192 in OpensslCipher.

- Adds equivalent support for 192-bit AES/CTR/NoPadding codec in the 
OpensslCipher
- Adds test-cases to cover 192-bit (24-bytes) and 256-bit (32-bytes) keys 
to both JCE and OpenSSL crypto tests
- Enhances the error message when an invalid Key or IV size is passed into 
OpensslCipher

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/QwertyManiac/hadoop HDFS-9878

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/135.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #135


commit 4e08bafaec31f0190f731f6ca0e07c7f2125d489
Author: Harsh J 
Date:   2016-10-06T05:52:22Z

HDFS-9878. Add support for AES-192 in OpenSSL native Cipher.




> Data transfer encryption with AES 192: Invalid key length.
> --
>
> Key: HDFS-9878
> URL: https://issues.apache.org/jira/browse/HDFS-9878
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode, hdfs-client
>Affects Versions: 2.7.2
> Environment: OS: Ubuntu 14.04
> /hadoop-2.7.2/bin$ uname -a
> Linux wkstn-kpalaniappan 3.13.0-79-generic #123-Ubuntu SMP Fri Feb 19 
> 14:27:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> /hadoop-2.7.2/bin$ java -version
> java version "1.7.0_95"
> OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-0ubuntu0.14.04.1)
> OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
> Hadoop version: 2.7.2
>Reporter: Karthik Palaniappan
>
> Configuring aes 128 or aes 256 encryption 
> (dfs.encrypt.data.transfer.cipher.key.bitlength = [128, 256]) works perfectly 
> fine. Trying to use AES 192 generates this exception on the datanode:
> 16/02/29 17:34:10 ERROR datanode.DataNode: 
> wkstn-kpalaniappan:50010:DataXceiver error processing unknown operation  src: 
> /127.0.0.1:57237 dst: /127.0.0.1:50010
> java.lang.IllegalArgumentException: Invalid key length.
>   at org.apache.hadoop.crypto.OpensslCipher.init(Native Method)
>   at org.apache.hadoop.crypto.OpensslCipher.init(OpensslCipher.java:176)
>   at 
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec$OpensslAesCtrCipher.init(OpensslAesCtrCryptoCodec.java:116)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.updateDecryptor(CryptoInputStream.java:290)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.resetStreamOffset(CryptoInputStream.java:303)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:128)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:109)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:133)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.createStreamPair(DataTransferSaslUtil.java:345)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:396)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:110)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:193)
>   at java.lang.Thread.run(Thread.java:745)
> And this exception on the client:
> /hadoop-2.7.2/bin$ ./hdfs dfs -copyFromLocal ~/.vimrc /vimrc
> 16/02/29 17:34:10 WARN hdfs.DFSClient: DataStreamer Exception
> java.lang.IllegalArgumentException: Invalid key length.
>   at org.apache.hadoop.crypto.OpensslCipher.init(Native Method)
>   at org.apache.hadoop.crypto.OpensslCipher.init(OpensslCipher.java:176)
>   at 
> org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec$OpensslAesCtrCipher.init(OpensslAesCtrCryptoCodec.java:116)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.updateDecryptor(CryptoInputStream.java:290)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.resetStreamOffset(CryptoInputStream.java:303)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:128)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:109)
>   at 
> org.apache.hadoop.crypto.CryptoInputStream.(CryptoInputStream.java:133)
>   at 
> 

[jira] [Updated] (HDFS-9337) Should check required params in WebHDFS to avoid NPE

[Jagadesh Kiran N (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jagadesh Kiran N updated HDFS-9337:
---
Attachment: HDFS-9337_06.patch

Updating with check style fix,pls review

> Should check required params in WebHDFS to avoid NPE
> 
>
> Key: HDFS-9337
> URL: https://issues.apache.org/jira/browse/HDFS-9337
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Jagadesh Kiran N
>Assignee: Jagadesh Kiran N
> Attachments: HDFS-9337_00.patch, HDFS-9337_01.patch, 
> HDFS-9337_02.patch, HDFS-9337_03.patch, HDFS-9337_04.patch, 
> HDFS-9337_05.patch, HDFS-9337_06.patch
>
>
> {code}
>  curl -i -X PUT 
> "http://10.19.92.127:50070/webhdfs/v1/kiran/sreenu?op=RENAMESNAPSHOT=SNAPSHOTNAME;
> {code}
> Null point exception will be thrown
> {code}
> {"RemoteException":{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException","message":null}}
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org