[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Xiaoyu Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525167#comment-15525167
 ] 

Xiaoyu Yao commented on HDFS-10690:
---

Thanks [~fenghua_hu] for updating the patch. +1 for patch v6 pending Jenkins. 

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> HDFS-10690.003.patch, HDFS-10690.004.patch, HDFS-10690.005.patch, 
> HDFS-10690.006.patch, ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10906) Add unit tests for Trash with HDFS encryption zones

[Xiaoyu Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525143#comment-15525143
 ] 

Xiaoyu Yao edited comment on HDFS-10906 at 9/27/16 5:50 AM:


Below is a proposed test plan for list of test cases to add:

# Trash Checkpoint within the encryption zone
# Trash Checkpoint outside the encryption zone when the whole encryption is 
deleted and moved.
# Trash expunge within the encryption zone
# Trash expunge outside the encryption zone
# Delete file/directory within the encryption zone (Kerberos off)
# Delete file/directory within the encryption zone (Kerberos on)
# Delete directory with/wo -skipTrash option
# Delete empty directory without -skipTrash option
# Delete entire encryption zone with different test users
# Delete file/directory from Trash within the encryption zone
# Namenode restart after Delete file/directory within the encryption zone



was (Author: xyao):
Below is a proposed test plan for list of test cases to add:

# Trash Checkpoint within the encryption zone
# Trash Checkpoint outside the encryption zone when the whole encryption is 
deleted and moved.
# Trash expunge within the encryption zone
# Trash expunge outside the encryption zone
# Delete file/directory within the encryption zone (Kerberos off)
# Delete file/directory within the encryption zone (Kerberos on)
# Delete directory with/wo -skipTrash option
# Delete empty directory without -skipTrash option
# Delete entire encryption zone with different test users
# Delete file/directory from Trash within the encryption zone


> Add unit tests for Trash with HDFS encryption zones
> ---
>
> Key: HDFS-10906
> URL: https://issues.apache.org/jira/browse/HDFS-10906
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: encryption
>Affects Versions: 2.8.0
>Reporter: Xiaoyu Yao
>
> The goal is to improve unit test coverage for HDFS trash with encryption zone 
> especially under Kerberos environment. The current unit test 
> TestEncryptionZones#testEncryptionZonewithTrash() has limited coverage on 
> non-Kerberos case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10906) Add unit tests for Trash with HDFS encryption zones

[Xiaoyu Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525143#comment-15525143
 ] 

Xiaoyu Yao commented on HDFS-10906:
---

Below is a proposed test plan for list of test cases to add:

# Trash Checkpoint within the encryption zone
# Trash Checkpoint outside the encryption zone when the whole encryption is 
deleted and moved.
# Trash expunge within the encryption zone
# Trash expunge outside the encryption zone
# Delete file/directory within the encryption zone (Kerberos off)
# Delete file/directory within the encryption zone (Kerberos on)
# Delete directory with/wo -skipTrash option
# Delete empty directory without -skipTrash option
# Delete entire encryption zone with different test users
# Delete file/directory from Trash within the encryption zone


> Add unit tests for Trash with HDFS encryption zones
> ---
>
> Key: HDFS-10906
> URL: https://issues.apache.org/jira/browse/HDFS-10906
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: encryption
>Affects Versions: 2.8.0
>Reporter: Xiaoyu Yao
>
> The goal is to improve unit test coverage for HDFS trash with encryption zone 
> especially under Kerberos environment. The current unit test 
> TestEncryptionZones#testEncryptionZonewithTrash() has limited coverage on 
> non-Kerberos case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9885) Correct the distcp counters name while displaying counters

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525141#comment-15525141
 ] 

Hudson commented on HDFS-9885:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10494 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10494/])
HDFS-9885. Correct the distcp counters name while displaying counters. (brahma: 
rev e17a4970bea8213660bb6c550104783069153236)
* (add) 
hadoop-tools/hadoop-distcp/src/main/resources/org/apache/hadoop/tools/mapred/CopyMapper_Counter.properties


> Correct the distcp counters name while displaying counters
> --
>
> Key: HDFS-9885
> URL: https://issues.apache.org/jira/browse/HDFS-9885
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: distcp
>Affects Versions: 2.7.1
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 2.8.0, 2.9.0, 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-9885.001.patch, HDFS-9885.002.patch
>
>
> In distcp cmd output,
> hadoop distcp hdfs://NN1:port/file1 hdfs://NN2:port/file2
> 16/02/29 07:05:55 INFO tools.DistCp: DistCp job-id: job_1456729398560_0002
> 16/02/29 07:05:55 INFO mapreduce.Job: Running job: job_1456729398560_0002
> 16/02/29 07:06:01 INFO mapreduce.Job: Job job_1456729398560_0002 running in 
> uber mode : false
> 16/02/29 07:06:01 INFO mapreduce.Job: map 0% reduce 0%
> 16/02/29 07:06:06 INFO mapreduce.Job: map 100% reduce 0%
> 16/02/29 07:06:07 INFO mapreduce.Job: Job job_1456729398560_0002 completed 
> successfully
> ...
> ...
> File Input Format Counters
> Bytes Read=212
> File Output Format Counters
> Bytes Written=0{color:red} 
> org.apache.hadoop.tools.mapred.CopyMapper$Counter
> {color}
> BANDWIDTH_IN_BYTES=12418
> BYTESCOPIED=12418
> BYTESEXPECTED=12418
> COPY=1
> Expected:
> Display Name can be given instead of 
> {color:red}"org.apache.hadoop.tools.mapred.CopyMapper$Counter"{color}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10906) Add unit tests for Trash with HDFS encryption zones

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDFS-10906:
-

 Summary: Add unit tests for Trash with HDFS encryption zones
 Key: HDFS-10906
 URL: https://issues.apache.org/jira/browse/HDFS-10906
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: encryption
Affects Versions: 2.8.0
Reporter: Xiaoyu Yao


The goal is to improve unit test coverage for HDFS trash with encryption zone 
especially under Kerberos environment. The current unit test 
TestEncryptionZones#testEncryptionZonewithTrash() has limited coverage on 
non-Kerberos case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9885) Correct the distcp counters name while displaying counters

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HDFS-9885:
---
   Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
   2.7.4
   2.9.0
   2.8.0
   Status: Resolved  (was: Patch Available)

Committed to trunk,branch-2,branch-2.8 and branch-2.7.. Thanks to [~archanat] 
for reporting,[~surendrasingh] for contribution..And thanks to [~yzhangal] and 
[~jzhuge] for additional revirew

> Correct the distcp counters name while displaying counters
> --
>
> Key: HDFS-9885
> URL: https://issues.apache.org/jira/browse/HDFS-9885
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: distcp
>Affects Versions: 2.7.1
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 2.8.0, 2.9.0, 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-9885.001.patch, HDFS-9885.002.patch
>
>
> In distcp cmd output,
> hadoop distcp hdfs://NN1:port/file1 hdfs://NN2:port/file2
> 16/02/29 07:05:55 INFO tools.DistCp: DistCp job-id: job_1456729398560_0002
> 16/02/29 07:05:55 INFO mapreduce.Job: Running job: job_1456729398560_0002
> 16/02/29 07:06:01 INFO mapreduce.Job: Job job_1456729398560_0002 running in 
> uber mode : false
> 16/02/29 07:06:01 INFO mapreduce.Job: map 0% reduce 0%
> 16/02/29 07:06:06 INFO mapreduce.Job: map 100% reduce 0%
> 16/02/29 07:06:07 INFO mapreduce.Job: Job job_1456729398560_0002 completed 
> successfully
> ...
> ...
> File Input Format Counters
> Bytes Read=212
> File Output Format Counters
> Bytes Written=0{color:red} 
> org.apache.hadoop.tools.mapred.CopyMapper$Counter
> {color}
> BANDWIDTH_IN_BYTES=12418
> BYTESCOPIED=12418
> BYTESEXPECTED=12418
> COPY=1
> Expected:
> Display Name can be given instead of 
> {color:red}"org.apache.hadoop.tools.mapred.CopyMapper$Counter"{color}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525118#comment-15525118
 ] 

Hadoop QA commented on HDFS-10713:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
49s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
17s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
7s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
43s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 29s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 203 unchanged - 3 fixed = 204 total (was 206) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 49m 
31s{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_111. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}146m 16s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HDFS-10713 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830430/HDFS-10713.branch-2.000.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 31dc7abb6ef4 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2 / 809a45a |
| Default Java | 1.7.0_111 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_101 

[jira] [Updated] (HDFS-9885) Correct the distcp counters name while displaying counters

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HDFS-9885:
---
Summary: Correct the distcp counters name while displaying counters  (was: 
Correct the distcp counters name wh)

> Correct the distcp counters name while displaying counters
> --
>
> Key: HDFS-9885
> URL: https://issues.apache.org/jira/browse/HDFS-9885
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: distcp
>Affects Versions: 2.7.1
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Attachments: HDFS-9885.001.patch, HDFS-9885.002.patch
>
>
> In distcp cmd output,
> hadoop distcp hdfs://NN1:port/file1 hdfs://NN2:port/file2
> 16/02/29 07:05:55 INFO tools.DistCp: DistCp job-id: job_1456729398560_0002
> 16/02/29 07:05:55 INFO mapreduce.Job: Running job: job_1456729398560_0002
> 16/02/29 07:06:01 INFO mapreduce.Job: Job job_1456729398560_0002 running in 
> uber mode : false
> 16/02/29 07:06:01 INFO mapreduce.Job: map 0% reduce 0%
> 16/02/29 07:06:06 INFO mapreduce.Job: map 100% reduce 0%
> 16/02/29 07:06:07 INFO mapreduce.Job: Job job_1456729398560_0002 completed 
> successfully
> ...
> ...
> File Input Format Counters
> Bytes Read=212
> File Output Format Counters
> Bytes Written=0{color:red} 
> org.apache.hadoop.tools.mapred.CopyMapper$Counter
> {color}
> BANDWIDTH_IN_BYTES=12418
> BYTESCOPIED=12418
> BYTESEXPECTED=12418
> COPY=1
> Expected:
> Display Name can be given instead of 
> {color:red}"org.apache.hadoop.tools.mapred.CopyMapper$Counter"{color}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9885) Correct the distcp counters name wh

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HDFS-9885:
---
Summary: Correct the distcp counters name wh  (was: In distcp cmd ouput, 
Display name should be given for 
org.apache.hadoop.tools.mapred.CopyMapper$Counter.)

> Correct the distcp counters name wh
> ---
>
> Key: HDFS-9885
> URL: https://issues.apache.org/jira/browse/HDFS-9885
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: distcp
>Affects Versions: 2.7.1
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Attachments: HDFS-9885.001.patch, HDFS-9885.002.patch
>
>
> In distcp cmd output,
> hadoop distcp hdfs://NN1:port/file1 hdfs://NN2:port/file2
> 16/02/29 07:05:55 INFO tools.DistCp: DistCp job-id: job_1456729398560_0002
> 16/02/29 07:05:55 INFO mapreduce.Job: Running job: job_1456729398560_0002
> 16/02/29 07:06:01 INFO mapreduce.Job: Job job_1456729398560_0002 running in 
> uber mode : false
> 16/02/29 07:06:01 INFO mapreduce.Job: map 0% reduce 0%
> 16/02/29 07:06:06 INFO mapreduce.Job: map 100% reduce 0%
> 16/02/29 07:06:07 INFO mapreduce.Job: Job job_1456729398560_0002 completed 
> successfully
> ...
> ...
> File Input Format Counters
> Bytes Read=212
> File Output Format Counters
> Bytes Written=0{color:red} 
> org.apache.hadoop.tools.mapred.CopyMapper$Counter
> {color}
> BANDWIDTH_IN_BYTES=12418
> BYTESCOPIED=12418
> BYTESEXPECTED=12418
> COPY=1
> Expected:
> Display Name can be given instead of 
> {color:red}"org.apache.hadoop.tools.mapred.CopyMapper$Counter"{color}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10828) Fix usage of FsDatasetImpl object lock in ReplicaMap

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525081#comment-15525081
 ] 

Anu Engineer commented on HDFS-10828:
-

[~arpitagarwal] Thanks for the patch. +1, LGTM. 

> Fix usage of FsDatasetImpl object lock in ReplicaMap
> 
>
> Key: HDFS-10828
> URL: https://issues.apache.org/jira/browse/HDFS-10828
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Blocker
> Attachments: HDFS-10828.01.patch, HDFS-10828.02.patch, 
> HDFS-10828.03.patch
>
>
> HDFS-10682 replaced the FsDatasetImpl object lock with a separate reentrant 
> lock but missed updating an instance ReplicaMap still uses the FsDatasetImpl.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9482) Replace DatanodeInfo constructors with a builder pattern

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HDFS-9482:
---
Attachment: HDFS-9482-002.patch

Uploaded the patch to fix applicable checksystyle and findbugs

> Replace DatanodeInfo constructors with a builder pattern
> 
>
> Key: HDFS-9482
> URL: https://issues.apache.org/jira/browse/HDFS-9482
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9482-002.patch, HDFS-9482.patch
>
>
> As per  [~arpitagarwal] comment 
> [here|https://issues.apache.org/jira/browse/HDFS-9038?focusedCommentId=15018761=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15018761],Replace
>  DatanodeInfo constructors with a builder pattern, 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9850) DiskBalancer : Explore removing references to FsVolumeSpi

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525002#comment-15525002
 ] 

Hadoop QA commented on HDFS-9850:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 44s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 32s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.mover.TestStorageMover |
|   | hadoop.hdfs.TestDecommissionWithStriped |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-9850 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830432/HDFS-9850.003.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 6ecaa6ee4eb6 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / edf0d0f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16874/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16874/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16874/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> DiskBalancer : Explore removing references to FsVolumeSpi 
> --
>
> Key: HDFS-9850
> URL: https://issues.apache.org/jira/browse/HDFS-9850
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Affects Versions: 3.0.0-alpha2
>  

[jira] [Comment Edited] (HDFS-9850) DiskBalancer : Explore removing references to FsVolumeSpi

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524963#comment-15524963
 ] 

Anu Engineer edited comment on HDFS-9850 at 9/27/16 3:59 AM:
-

[~manojg] Thanks for quick turnaround. I was reading code and something struck 
me. Could you please help me understand what would happen in the following 
scenario ? 

Let us say a plan was executed, then either the source or destination disk was 
removed/reconfigured and then I call into queryStatus, 
Would it work or would it fail ? 





was (Author: anu):
[~manojg] Thanks for quick turnaround. I was reading code and something struck 
me. Could you please help me understand what would happen in the following 
scenario ? 

Let us say a plan was executed, then either the source or destination disk was 
removed/reconfigured and then I call into queryStatus ? 
Would it work or would it fail ? 




> DiskBalancer : Explore removing references to FsVolumeSpi 
> --
>
> Key: HDFS-9850
> URL: https://issues.apache.org/jira/browse/HDFS-9850
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Affects Versions: 3.0.0-alpha2
>Reporter: Anu Engineer
>Assignee: Manoj Govindassamy
> Attachments: HDFS-9850.001.patch, HDFS-9850.002.patch, 
> HDFS-9850.003.patch
>
>
> In HDFS-9671, [~arpitagarwal] commented that we should explore the 
> possibility of removing references to FsVolumeSpi at any point and only deal 
> with storage ID. We are not sure if this is possible, this JIRA is to explore 
> if that can be done without issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10824) MiniDFSCluster#storageCapacities has no effects on real capacity

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524980#comment-15524980
 ] 

Xiaobing Zhou commented on HDFS-10824:
--

The test failure is not related to the patch. It actually passed with latest 
code in my local.

> MiniDFSCluster#storageCapacities has no effects on real capacity
> 
>
> Key: HDFS-10824
> URL: https://issues.apache.org/jira/browse/HDFS-10824
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10824.000.patch, HDFS-10824.001.patch, 
> HDFS-10824.002.patch, HDFS-10824.003.patch, HDFS-10824.004.patch, 
> HDFS-10824.005.patch
>
>
> It has been noticed MiniDFSCluster#storageCapacities has no effects on real 
> capacity. It can be reproduced by explicitly setting storageCapacities and 
> then call ClientProtocol#getDatanodeStorageReport(DatanodeReportType.LIVE) to 
> compare results. The following are  storage report for one node with two 
> volumes after I set capacity as 300 * 1024. Apparently, the capacity is not 
> changed.
> adminState|DatanodeInfo$AdminStates  (id=6861)
> |blockPoolUsed|215192|
> |cacheCapacity|0|
> |cacheUsed|0|
> |capacity|998164971520|
> |datanodeUuid|"839912e9-5bcb-45d1-81cf-9a9c9c02a00b" (id=6862)|
> |dependentHostNames|LinkedList  (id=6863)|
> |dfsUsed|215192|
> |hostName|"127.0.0.1" (id=6864)|
> |infoPort|64222|
> |infoSecurePort|0|
> |ipAddr|"127.0.0.1" (id=6865)|
> |ipcPort|64223|
> |lastUpdate|1472682790948|
> |lastUpdateMonotonic|209605640|
> |level|0|
> |location|"/default-rack" (id=6866)|
> |maintenanceExpireTimeInMS|0|
> |parent|null|
> |peerHostName|null|
> |remaining|20486512640|
> |softwareVersion|null|
> |upgradeDomain|null|
> |xceiverCount|1|
> |xferAddr|"127.0.0.1:64220" (id=6855)|
> |xferPort|64220|
> [0]StorageReport  (id=6856)
> |blockPoolUsed|4096|
> |capacity|499082485760|
> |dfsUsed|4096|
> |failed|false|
> |remaining|10243256320|
> |storage|DatanodeStorage  (id=6869)|
> [1]StorageReport  (id=6859)
> |blockPoolUsed|211096|
> |capacity|499082485760|
> |dfsUsed|211096|
> |failed|false|
> |remaining|10243256320|
> |storage|DatanodeStorage  (id=6872)|



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order

[Arpit Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524968#comment-15524968
 ] 

Arpit Agarwal commented on HDFS-10301:
--

bq.  As all storage reports in the single RPC BR satisfy the condition that 
triggers removal of the lease, all storage reports after the first storage 
report will be ignored without the change.
Isn't that because the patch also removes the lastStorageInRpc check? If that 
check was not removed then the workaround wouldn't be necessary.

bq. When BRs are split into multiple RPCS: Say 2 BRs from the same DN are 
processed at the same time.
We should make a more resilient fix that doesn't require the lease ID 
workaround. In the interests of making forward progress, can we just remove the 
zombie processing for now and fix the other issues separately?

> BlockReport retransmissions may lead to storages falsely being declared 
> zombie if storage report processing happens out of order
> 
>
> Key: HDFS-10301
> URL: https://issues.apache.org/jira/browse/HDFS-10301
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.6.1
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
>Priority: Critical
> Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, 
> HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.006.patch, 
> HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch, 
> HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch, 
> HDFS-10301.012.patch, HDFS-10301.013.patch, HDFS-10301.014.patch, 
> HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch, 
> HDFS-10301.sample.patch, zombieStorageLogs.rtf
>
>
> When NameNode is busy a DataNode can timeout sending a block report. Then it 
> sends the block report again. Then NameNode while process these two reports 
> at the same time can interleave processing storages from different reports. 
> This screws up the blockReportId field, which makes NameNode think that some 
> storages are zombie. Replicas from zombie storages are immediately removed, 
> causing missing blocks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9850) DiskBalancer : Explore removing references to FsVolumeSpi

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524963#comment-15524963
 ] 

Anu Engineer commented on HDFS-9850:


[~manojg] Thanks for quick turnaround. I was reading code and something struck 
me. Could you please help me understand what would happen in the following 
scenario ? 

Let us say a plan was executed, then either the source or destination disk was 
removed/reconfigured and then I call into queryStatus ? 
Would it work or would it fail ? 




> DiskBalancer : Explore removing references to FsVolumeSpi 
> --
>
> Key: HDFS-9850
> URL: https://issues.apache.org/jira/browse/HDFS-9850
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Affects Versions: 3.0.0-alpha2
>Reporter: Anu Engineer
>Assignee: Manoj Govindassamy
> Attachments: HDFS-9850.001.patch, HDFS-9850.002.patch, 
> HDFS-9850.003.patch
>
>
> In HDFS-9671, [~arpitagarwal] commented that we should explore the 
> possibility of removing references to FsVolumeSpi at any point and only deal 
> with storage ID. We are not sure if this is possible, this JIRA is to explore 
> if that can be done without issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings

[Arpit Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524943#comment-15524943
 ] 

Arpit Agarwal commented on HDFS-10713:
--

+1 for the branch-2 patch, pending Jenkins.

> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, 
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, 
> HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, 
> HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, 
> HDFS-10713.011.patch, HDFS-10713.012.patch, HDFS-10713.branch-2.000.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a 
> thread for over 1 second. These messages can be throttled to at one most one 
> per x minutes to avoid potentially filling up NN logs. We can also log the 
> number of suppressed notices since the last log message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9444) Add utility to find set of available ephemeral ports to ServerSocketUtil

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524878#comment-15524878
 ] 

Hudson commented on HDFS-9444:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10492 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10492/])
HDFS-9444. Add utility to find set of available ephemeral ports to (brahma: rev 
edf0d0f8b2115d4edb5d4932b5ecb15430d94c40)
* (edit) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestEditLogTailer.java


> Add utility to find set of available ephemeral ports to ServerSocketUtil
> 
>
> Key: HDFS-9444
> URL: https://issues.apache.org/jira/browse/HDFS-9444
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: test
>Reporter: Brahma Reddy Battula
>Assignee: Masatake Iwasaki
> Attachments: HDFS-9444.001.patch, HDFS-9444.002.patch, 
> HDFS-9444.003.patch, HDFS-9444.004.patch, HDFS-9444.005.patch, 
> HDFS-9444.006.patch
>
>
> Unit tests using MiniDFSCluster with namanode-ha enabled need set of port 
> numbers in advance. Because namenodes talk to each other, we can not set ipc 
> port to 0 in configuration to make namenodes decide port number on its own. 
> ServerSocketUtil should provide utility to find set of available ephemeral 
> port numbers for this.
> For example, TestEditLogTailer could fail due to {{java.net.BindException: 
> Address already in use}}.
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/2556/testReport/
> {noformat}
> java.net.BindException: Problem binding to [localhost:42477] 
> java.net.BindException: Address already in use; For more details see:  
> http://wiki.apache.org/hadoop/BindException
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:444)
>   at sun.nio.ch.Net.bind(Net.java:436)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at org.apache.hadoop.ipc.Server.bind(Server.java:469)
>   at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
>   at org.apache.hadoop.ipc.Server.(Server.java:2464)
>   at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
>   at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRolls(TestEditLogTailer.java:139)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testNN1TriggersLogRolls(TestEditLogTailer.java:114)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10824) MiniDFSCluster#storageCapacities has no effects on real capacity

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524856#comment-15524856
 ] 

Hadoop QA commented on HDFS-10824:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 27s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 205 unchanged - 1 fixed = 206 total (was 206) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 44s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 36s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10824 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830427/HDFS-10824.005.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 76504fbe1d1c 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 059058f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16873/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16873/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16873/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16873/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> MiniDFSCluster#storageCapacities has no effects on real capacity
> 
>
> Key: 

[jira] [Assigned] (HDFS-10905) Refactor DataStreamer#createBlockOutputStream

[Yuanbo Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuanbo Liu reassigned HDFS-10905:
-

Assignee: Yuanbo Liu

> Refactor DataStreamer#createBlockOutputStream
> -
>
> Key: HDFS-10905
> URL: https://issues.apache.org/jira/browse/HDFS-10905
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Yuanbo Liu
>Priority: Minor
>
> DataStreamer#createBlockOutputStream and DataStreamer#transfer shared much 
> boilerplate code. HDFS-10609 refactored the transfer method into a 
> StreamerStreams class. The createBlockOutputStream method should reuse the 
> class to de-dup code and to improve code clarity.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10892) Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524824#comment-15524824
 ] 

Hadoop QA commented on HDFS-10892:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 24s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 2 new + 179 unchanged - 0 fixed = 181 total (was 179) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 59s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m  6s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDecommissionWithStriped |
|   | hadoop.hdfs.TestDFSShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10892 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830424/HDFS-10892.003.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 27b7050f80ea 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 059058f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16872/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16872/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16872/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16872/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'
> ---
>
> Key: 

[jira] [Updated] (HDFS-9895) Remove unnecessary conf cache from DataNode

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-9895:

   Resolution: Fixed
Fix Version/s: 2.9.0
   Status: Resolved  (was: Patch Available)

Thanks [~xiaobingo]. Pushed to branch-2.

> Remove unnecessary conf cache from DataNode
> ---
>
> Key: HDFS-9895
> URL: https://issues.apache.org/jira/browse/HDFS-9895
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: HDFS-9895-HDFS-9000.002.patch, 
> HDFS-9895-HDFS-9000.003.patch, HDFS-9895-branch-2.003.patch, 
> HDFS-9895.000.patch, HDFS-9895.001.patch
>
>
> Since DataNode inherits ReconfigurableBase with Configured as base class 
> where configuration is maintained, DataNode#conf should be removed for the 
> purpose of brevity.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10804) Use separate lock for ReplicaMap

[Fenghua Hu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fenghua Hu updated HDFS-10804:
--
Summary: Use separate lock for ReplicaMap  (was: Use finer-granularity lock 
for ReplicaMap)

> Use separate lock for ReplicaMap
> 
>
> Key: HDFS-10804
> URL: https://issues.apache.org/jira/browse/HDFS-10804
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0-beta1
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
>Priority: Minor
> Attachments: HDFS-10804-001.patch, HDFS-10804-002.patch
>
>
> In currently implementation, ReplicaMap takes an external object as lock for 
> synchronization.
> In function FsDatasetImpl#FsDatasetImpl(), the object is for synchronization 
> is "this", i.e. FsDatasetImpl: 
> volumeMap = new ReplicaMap(this);
> and in private FsDatasetImpl#addVolume(), "this" object is used for 
> synchronization as well.
> ReplicaMap tempVolumeMap = new ReplicaMap(this);
> I am not sure if we really need so big object FsDatasetImpl  for ReplicaMap's 
> synchronization. If it's not necessary, this could reduce lock contention on 
> FsDatasetImpl object and improve performance. 
> Could you please give me some suggestions? Thanks a lot!
> Fenghua



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10828) Fix usage of FsDatasetImpl object lock in ReplicaMap

[Fenghua Hu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524786#comment-15524786
 ] 

Fenghua Hu commented on HDFS-10828:
---

[~arpitagarwal], 

Actually, i opened another JIRA 
https://issues.apache.org/jira/browse/HDFS-10804 for replicaMap lock issue, but 
after seeing your fix, i thought we could consider them together in HDFS-10828. 

I agree with you that it makes sense to keep the existing behavior in this fix, 
and we will use HDFS-10804 to track ReplicaMap issue.  Thanks.

> Fix usage of FsDatasetImpl object lock in ReplicaMap
> 
>
> Key: HDFS-10828
> URL: https://issues.apache.org/jira/browse/HDFS-10828
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Blocker
> Attachments: HDFS-10828.01.patch, HDFS-10828.02.patch, 
> HDFS-10828.03.patch
>
>
> HDFS-10682 replaced the FsDatasetImpl object lock with a separate reentrant 
> lock but missed updating an instance ReplicaMap still uses the FsDatasetImpl.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8312) Trash does not descent into child directories to check for permissions

[Weiwei Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524781#comment-15524781
 ] 

Weiwei Yang commented on HDFS-8312:
---

Hello [~daryn]

bq. The NN should not be deleting directories as the super-user on behalf of 
non-privileged users. 

I agree with that. But the permission should be checked before moving the 
files/dirs to trash. If you use the original rename to move the files/dirs to 
trash, after a while, deletion is denied due to insufficient permission, that 
looks like odd.

> Trash does not descent into child directories to check for permissions
> --
>
> Key: HDFS-8312
> URL: https://issues.apache.org/jira/browse/HDFS-8312
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs, security
>Affects Versions: 2.2.0, 2.6.0, 2.7.2
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: HDFS-8312-001.patch, HDFS-8312-002.patch, 
> HDFS-8312-003.patch, HDFS-8312-004.patch, HDFS-8312-005.patch, 
> HDFS-8312-testcase.patch
>
>
> HDFS trash does not descent into child directory to check if user has 
> permission to delete files.  For example:
> Run the following command to initialize directory structure as super user:
> {code}
> hadoop fs -mkdir /BSS/level1
> hadoop fs -mkdir /BSS/level1/level2
> hadoop fs -mkdir /BSS/level1/level2/level3
> hadoop fs -put /tmp/appConfig.json /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chown user1:users /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chown -R user1:users /BSS/level1
> hadoop fs -chown -R 750 /BSS/level1
> hadoop fs -chmod -R 640 /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chmod 775 /BSS
> {code}
> Change to a normal user called user2. 
> When trash is enabled:
> {code}
> sudo su user2 -
> hadoop fs -rm -r /BSS/level1
> 15/05/01 16:51:20 INFO fs.TrashPolicyDefault: Namenode trash configuration: 
> Deletion interval = 3600 minutes, Emptier interval = 0 minutes.
> Moved: 'hdfs://bdvs323.svl.ibm.com:9000/BSS/level1' to trash at: 
> hdfs://bdvs323.svl.ibm.com:9000/user/user2/.Trash/Current
> {code}
> When trash is disabled:
> {code}
> /opt/ibm/biginsights/IHC/bin/hadoop fs -Dfs.trash.interval=0 -rm -r 
> /BSS/level1
> 15/05/01 16:58:31 INFO fs.TrashPolicyDefault: Namenode trash configuration: 
> Deletion interval = 0 minutes, Emptier interval = 0 minutes.
> rm: Permission denied: user=user2, access=ALL, 
> inode="/BSS/level1":user1:users:drwxr-x---
> {code}
> There is inconsistency between trash behavior and delete behavior.  When 
> trash is enabled, files owned by user1 is deleted by user2.  It looks like 
> trash does not recursively validate if the child directory files can be 
> removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10828) Fix usage of FsDatasetImpl object lock in ReplicaMap

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-10828:
-
Target Version/s: 2.8.0, 3.0.0-alpha2  (was: 2.8.0)

> Fix usage of FsDatasetImpl object lock in ReplicaMap
> 
>
> Key: HDFS-10828
> URL: https://issues.apache.org/jira/browse/HDFS-10828
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Blocker
> Attachments: HDFS-10828.01.patch, HDFS-10828.02.patch, 
> HDFS-10828.03.patch
>
>
> HDFS-10682 replaced the FsDatasetImpl object lock with a separate reentrant 
> lock but missed updating an instance ReplicaMap still uses the FsDatasetImpl.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10828) Fix usage of FsDatasetImpl object lock in ReplicaMap

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-10828:
-
Priority: Blocker  (was: Major)

> Fix usage of FsDatasetImpl object lock in ReplicaMap
> 
>
> Key: HDFS-10828
> URL: https://issues.apache.org/jira/browse/HDFS-10828
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Blocker
> Attachments: HDFS-10828.01.patch, HDFS-10828.02.patch, 
> HDFS-10828.03.patch
>
>
> HDFS-10682 replaced the FsDatasetImpl object lock with a separate reentrant 
> lock but missed updating an instance ReplicaMap still uses the FsDatasetImpl.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10828) Fix usage of FsDatasetImpl object lock in ReplicaMap

[Arpit Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524766#comment-15524766
 ] 

Arpit Agarwal commented on HDFS-10828:
--

Hi [~fenghua_hu], that probably makes sense although we may need more analysis 
to ensure it doesn't introduce synchronization issues. 

This change is meant to fix the immediate regression and we can certainly 
consider 

> Fix usage of FsDatasetImpl object lock in ReplicaMap
> 
>
> Key: HDFS-10828
> URL: https://issues.apache.org/jira/browse/HDFS-10828
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-10828.01.patch, HDFS-10828.02.patch, 
> HDFS-10828.03.patch
>
>
> HDFS-10682 replaced the FsDatasetImpl object lock with a separate reentrant 
> lock but missed updating an instance ReplicaMap still uses the FsDatasetImpl.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10828) Fix usage of FsDatasetImpl object lock in ReplicaMap

[Arpit Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524766#comment-15524766
 ] 

Arpit Agarwal edited comment on HDFS-10828 at 9/27/16 1:54 AM:
---

Hi [~fenghua_hu], that probably makes sense although we may need more analysis 
to ensure it doesn't introduce synchronization issues. 

This change is meant to fix the immediate regression and keep the behavior 
consistent with what we had earlier. 


was (Author: arpitagarwal):
Hi [~fenghua_hu], that probably makes sense although we may need more analysis 
to ensure it doesn't introduce synchronization issues. 

This change is meant to fix the immediate regression and we can certainly 
consider 

> Fix usage of FsDatasetImpl object lock in ReplicaMap
> 
>
> Key: HDFS-10828
> URL: https://issues.apache.org/jira/browse/HDFS-10828
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-10828.01.patch, HDFS-10828.02.patch, 
> HDFS-10828.03.patch
>
>
> HDFS-10682 replaced the FsDatasetImpl object lock with a separate reentrant 
> lock but missed updating an instance ReplicaMap still uses the FsDatasetImpl.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Fenghua Hu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524750#comment-15524750
 ] 

Fenghua Hu commented on HDFS-10690:
---

Removed unnessary "if (eldestKey == null)" statement, and updated the patch.

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> HDFS-10690.003.patch, HDFS-10690.004.patch, HDFS-10690.005.patch, 
> HDFS-10690.006.patch, ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Fenghua Hu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fenghua Hu updated HDFS-10690:
--
Attachment: HDFS-10690.006.patch

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> HDFS-10690.003.patch, HDFS-10690.004.patch, HDFS-10690.005.patch, 
> HDFS-10690.006.patch, ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10713) Throttle FsNameSystem lock warnings

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-10713:
-
Status: Patch Available  (was: Reopened)

> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, 
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, 
> HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, 
> HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, 
> HDFS-10713.011.patch, HDFS-10713.012.patch, HDFS-10713.branch-2.000.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a 
> thread for over 1 second. These messages can be throttled to at one most one 
> per x minutes to avoid potentially filling up NN logs. We can also log the 
> number of suppressed notices since the last log message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-10713) Throttle FsNameSystem lock warnings

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal reopened HDFS-10713:
--

> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, 
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, 
> HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, 
> HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, 
> HDFS-10713.011.patch, HDFS-10713.012.patch, HDFS-10713.branch-2.000.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a 
> thread for over 1 second. These messages can be throttled to at one most one 
> per x minutes to avoid potentially filling up NN logs. We can also log the 
> number of suppressed notices since the last log message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9444) Add utility to find set of available ephemeral ports to ServerSocketUtil

[Brahma Reddy Battula (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524733#comment-15524733
 ] 

Brahma Reddy Battula commented on HDFS-9444:


Committed to trunk...[~iwasakims] patch will not directly apply branch-2, there 
is a conflict in {{TestEditLogTailer}} . can you upload branch-2 patch..

> Add utility to find set of available ephemeral ports to ServerSocketUtil
> 
>
> Key: HDFS-9444
> URL: https://issues.apache.org/jira/browse/HDFS-9444
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: test
>Reporter: Brahma Reddy Battula
>Assignee: Masatake Iwasaki
> Attachments: HDFS-9444.001.patch, HDFS-9444.002.patch, 
> HDFS-9444.003.patch, HDFS-9444.004.patch, HDFS-9444.005.patch, 
> HDFS-9444.006.patch
>
>
> Unit tests using MiniDFSCluster with namanode-ha enabled need set of port 
> numbers in advance. Because namenodes talk to each other, we can not set ipc 
> port to 0 in configuration to make namenodes decide port number on its own. 
> ServerSocketUtil should provide utility to find set of available ephemeral 
> port numbers for this.
> For example, TestEditLogTailer could fail due to {{java.net.BindException: 
> Address already in use}}.
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/2556/testReport/
> {noformat}
> java.net.BindException: Problem binding to [localhost:42477] 
> java.net.BindException: Address already in use; For more details see:  
> http://wiki.apache.org/hadoop/BindException
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:444)
>   at sun.nio.ch.Net.bind(Net.java:436)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at org.apache.hadoop.ipc.Server.bind(Server.java:469)
>   at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
>   at org.apache.hadoop.ipc.Server.(Server.java:2464)
>   at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
>   at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRolls(TestEditLogTailer.java:139)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testNN1TriggersLogRolls(TestEditLogTailer.java:114)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9850) DiskBalancer : Explore removing references to FsVolumeSpi

[Manoj Govindassamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manoj Govindassamy updated HDFS-9850:
-
Attachment: HDFS-9850.003.patch

Attaching v03 patch to address checkstyle and review comments from [~eddyxu] 
and [~anu]. Kindly take a look.

> DiskBalancer : Explore removing references to FsVolumeSpi 
> --
>
> Key: HDFS-9850
> URL: https://issues.apache.org/jira/browse/HDFS-9850
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Affects Versions: 3.0.0-alpha2
>Reporter: Anu Engineer
>Assignee: Manoj Govindassamy
> Attachments: HDFS-9850.001.patch, HDFS-9850.002.patch, 
> HDFS-9850.003.patch
>
>
> In HDFS-9671, [~arpitagarwal] commented that we should explore the 
> possibility of removing references to FsVolumeSpi at any point and only deal 
> with storage ID. We are not sure if this is possible, this JIRA is to explore 
> if that can be done without issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9850) DiskBalancer : Explore removing references to FsVolumeSpi

[Manoj Govindassamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524718#comment-15524718
 ] 

Manoj Govindassamy commented on HDFS-9850:
--

Thanks for the review and comments [~anu]. Will be uploading the comments 
incorporated patch soon.
1. Done. Made use of private method getFsVolume(), just like other places.
2. Done. Restored "Disk Balancer -" in the removed logs and in the new logs. 
Even if you don't have these explicit signature in the logging, they already 
carry file name from where the logging is happening and here in this case it 
will be DiskBalancer.java, which can also be used for grepping
3. Done. Restored Volume path in the error message. True volume path when 
available can be used for error logging for they are much more easier to read.
4. Done. Corrected the method comments.
5. Done. Removed the unnecessary setExitFlag() calls.
6. Done. Removed the newly introduced interface method. Rather implemented as a 
new private method in DiskBalancer using the already available 
getVolumeReferences().
7. Done. Added the assert for reconfigure error.
8. Done. Test now checks for the error message prefix also.

> DiskBalancer : Explore removing references to FsVolumeSpi 
> --
>
> Key: HDFS-9850
> URL: https://issues.apache.org/jira/browse/HDFS-9850
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Affects Versions: 3.0.0-alpha2
>Reporter: Anu Engineer
>Assignee: Manoj Govindassamy
> Attachments: HDFS-9850.001.patch, HDFS-9850.002.patch
>
>
> In HDFS-9671, [~arpitagarwal] commented that we should explore the 
> possibility of removing references to FsVolumeSpi at any point and only deal 
> with storage ID. We are not sure if this is possible, this JIRA is to explore 
> if that can be done without issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Fenghua Hu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524699#comment-15524699
 ] 

Fenghua Hu commented on HDFS-10690:
---

Thanks [~xyao] for your suggestion. How about we just remove "if (eldestKey == 
null) { break; } for bulletin 2? If so we won't need an extra helper function 
for bulletin 1.

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> HDFS-10690.003.patch, HDFS-10690.004.patch, HDFS-10690.005.patch, 
> ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10900) DiskBalancer: Complete the documents for the report command

[Yiqun Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524698#comment-15524698
 ] 

Yiqun Lin edited comment on HDFS-10900 at 9/27/16 1:18 AM:
---

Thanks [~anu] for the review. One comment from me:
{quote}
 I will commit this, but after filing another JIRA.
{quote}
What's the meaning of this? Does it means that I should crate another JIRA and 
upload the same patch in this JIRA? Can make a more concrete explainaton?


was (Author: linyiqun):
Thanks [~anu] for the review. One comment from me:
{quote}
 I will commit this, but after filing another JIRA.
{quote}
What's the meaning of this? Does it means that I should crate another JIRA and 
upload the same patch in this JIRA? Can make a more concrete explnation?

> DiskBalancer: Complete the documents for the report command
> ---
>
> Key: HDFS-10900
> URL: https://issues.apache.org/jira/browse/HDFS-10900
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10900.001.patch
>
>
> Now the documemnts of the command {{hdfs diskbalancer -report}} look not 
> completed. Two minors:
> * The usage of {{hdfs diskbalancer -report}} is missing in {{HDFSCommands.md}}
> * One subcommand  of report command {{hdfs diskbalancer -report -top}} is 
> missing in {{HDFSDiskBalancer.md}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10900) DiskBalancer: Complete the documents for the report command

[Yiqun Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524698#comment-15524698
 ] 

Yiqun Lin commented on HDFS-10900:
--

Thanks [~anu] for the review. One comment from me:
{quote}
 I will commit this, but after filing another JIRA.
{quote}
What's the meaning of this? Does it means that I should crate another JIRA and 
upload the same patch in this JIRA? Can make a more concrete explnation?

> DiskBalancer: Complete the documents for the report command
> ---
>
> Key: HDFS-10900
> URL: https://issues.apache.org/jira/browse/HDFS-10900
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10900.001.patch
>
>
> Now the documemnts of the command {{hdfs diskbalancer -report}} look not 
> completed. Two minors:
> * The usage of {{hdfs diskbalancer -report}} is missing in {{HDFSCommands.md}}
> * One subcommand  of report command {{hdfs diskbalancer -report -top}} is 
> missing in {{HDFSDiskBalancer.md}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524693#comment-15524693
 ] 

Hudson commented on HDFS-10713:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10491 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10491/])
HDFS-10713. Throttle FsNameSystem lock warnings. Contributed by Hanisha (arp: 
rev 059058f9614613667d5385f76022294e07e140aa)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java


> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, 
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, 
> HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, 
> HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, 
> HDFS-10713.011.patch, HDFS-10713.012.patch, HDFS-10713.branch-2.000.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a 
> thread for over 1 second. These messages can be throttled to at one most one 
> per x minutes to avoid potentially filling up NN logs. We can also log the 
> number of suppressed notices since the last log message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10897) Ozone: SCM: Add NodeManager

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524671#comment-15524671
 ] 

Hadoop QA commented on HDFS-10897:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
1s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
4s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
21s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
21s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  1m 
14s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 
unchanged - 0 fixed = 8 total (was 7) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 44s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 37s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestCrcCorruption |
|   | hadoop.hdfs.server.datanode.TestDataNodeLifeline |
|   | hadoop.hdfs.server.datanode.TestBlockPoolManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10897 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830408/HDFS-10897-HDFS-7240.003.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux cebf19c0e193 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | HDFS-7240 / 0f73cd7 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16870/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| javadoc | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16870/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16870/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16870/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 

[jira] [Commented] (HDFS-10896) Move lock logging logic from FSNamesystem into FSNamesystemLock

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524658#comment-15524658
 ] 

Hadoop QA commented on HDFS-10896:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 33s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 2 new + 585 unchanged - 6 fixed = 587 total (was 591) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  2m  
3s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 24s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
17s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 45s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
|  |  Format string should use %n rather than n in 
org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.readUnlock()  At 
FSNamesystemLock.java:rather than n in 
org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.readUnlock()  At 
FSNamesystemLock.java:[line 93] |
|  |  Format string should use %n rather than n in 
org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock()  At 
FSNamesystemLock.java:rather than n in 
org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock()  At 
FSNamesystemLock.java:[line 123] |
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
|   | hadoop.tools.TestHdfsConfigFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10896 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830403/HDFS-10896.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 87e203bc340e 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8e06d86 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 

[jira] [Updated] (HDFS-10713) Throttle FsNameSystem lock warnings

[Hanisha Koneru (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hanisha Koneru updated HDFS-10713:
--
Attachment: HDFS-10713.branch-2.000.patch

Thank you [~arpitagarwal] for reviewing and committing the patch. I have posted 
a patch for branch-2.

> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, 
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, 
> HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, 
> HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, 
> HDFS-10713.011.patch, HDFS-10713.012.patch, HDFS-10713.branch-2.000.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a 
> thread for over 1 second. These messages can be throttled to at one most one 
> per x minutes to avoid potentially filling up NN logs. We can also log the 
> number of suppressed notices since the last log message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10824) MiniDFSCluster#storageCapacities has no effects on real capacity

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524599#comment-15524599
 ] 

Xiaobing Zhou commented on HDFS-10824:
--

Thanks [~cnauroth] for review. v005 is posted by moving 
waitDataNodeFullyStarted.

> MiniDFSCluster#storageCapacities has no effects on real capacity
> 
>
> Key: HDFS-10824
> URL: https://issues.apache.org/jira/browse/HDFS-10824
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10824.000.patch, HDFS-10824.001.patch, 
> HDFS-10824.002.patch, HDFS-10824.003.patch, HDFS-10824.004.patch, 
> HDFS-10824.005.patch
>
>
> It has been noticed MiniDFSCluster#storageCapacities has no effects on real 
> capacity. It can be reproduced by explicitly setting storageCapacities and 
> then call ClientProtocol#getDatanodeStorageReport(DatanodeReportType.LIVE) to 
> compare results. The following are  storage report for one node with two 
> volumes after I set capacity as 300 * 1024. Apparently, the capacity is not 
> changed.
> adminState|DatanodeInfo$AdminStates  (id=6861)
> |blockPoolUsed|215192|
> |cacheCapacity|0|
> |cacheUsed|0|
> |capacity|998164971520|
> |datanodeUuid|"839912e9-5bcb-45d1-81cf-9a9c9c02a00b" (id=6862)|
> |dependentHostNames|LinkedList  (id=6863)|
> |dfsUsed|215192|
> |hostName|"127.0.0.1" (id=6864)|
> |infoPort|64222|
> |infoSecurePort|0|
> |ipAddr|"127.0.0.1" (id=6865)|
> |ipcPort|64223|
> |lastUpdate|1472682790948|
> |lastUpdateMonotonic|209605640|
> |level|0|
> |location|"/default-rack" (id=6866)|
> |maintenanceExpireTimeInMS|0|
> |parent|null|
> |peerHostName|null|
> |remaining|20486512640|
> |softwareVersion|null|
> |upgradeDomain|null|
> |xceiverCount|1|
> |xferAddr|"127.0.0.1:64220" (id=6855)|
> |xferPort|64220|
> [0]StorageReport  (id=6856)
> |blockPoolUsed|4096|
> |capacity|499082485760|
> |dfsUsed|4096|
> |failed|false|
> |remaining|10243256320|
> |storage|DatanodeStorage  (id=6869)|
> [1]StorageReport  (id=6859)
> |blockPoolUsed|211096|
> |capacity|499082485760|
> |dfsUsed|211096|
> |failed|false|
> |remaining|10243256320|
> |storage|DatanodeStorage  (id=6872)|



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10824) MiniDFSCluster#storageCapacities has no effects on real capacity

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10824:
-
Attachment: HDFS-10824.005.patch

> MiniDFSCluster#storageCapacities has no effects on real capacity
> 
>
> Key: HDFS-10824
> URL: https://issues.apache.org/jira/browse/HDFS-10824
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10824.000.patch, HDFS-10824.001.patch, 
> HDFS-10824.002.patch, HDFS-10824.003.patch, HDFS-10824.004.patch, 
> HDFS-10824.005.patch
>
>
> It has been noticed MiniDFSCluster#storageCapacities has no effects on real 
> capacity. It can be reproduced by explicitly setting storageCapacities and 
> then call ClientProtocol#getDatanodeStorageReport(DatanodeReportType.LIVE) to 
> compare results. The following are  storage report for one node with two 
> volumes after I set capacity as 300 * 1024. Apparently, the capacity is not 
> changed.
> adminState|DatanodeInfo$AdminStates  (id=6861)
> |blockPoolUsed|215192|
> |cacheCapacity|0|
> |cacheUsed|0|
> |capacity|998164971520|
> |datanodeUuid|"839912e9-5bcb-45d1-81cf-9a9c9c02a00b" (id=6862)|
> |dependentHostNames|LinkedList  (id=6863)|
> |dfsUsed|215192|
> |hostName|"127.0.0.1" (id=6864)|
> |infoPort|64222|
> |infoSecurePort|0|
> |ipAddr|"127.0.0.1" (id=6865)|
> |ipcPort|64223|
> |lastUpdate|1472682790948|
> |lastUpdateMonotonic|209605640|
> |level|0|
> |location|"/default-rack" (id=6866)|
> |maintenanceExpireTimeInMS|0|
> |parent|null|
> |peerHostName|null|
> |remaining|20486512640|
> |softwareVersion|null|
> |upgradeDomain|null|
> |xceiverCount|1|
> |xferAddr|"127.0.0.1:64220" (id=6855)|
> |xferPort|64220|
> [0]StorageReport  (id=6856)
> |blockPoolUsed|4096|
> |capacity|499082485760|
> |dfsUsed|4096|
> |failed|false|
> |remaining|10243256320|
> |storage|DatanodeStorage  (id=6869)|
> [1]StorageReport  (id=6859)
> |blockPoolUsed|211096|
> |capacity|499082485760|
> |dfsUsed|211096|
> |failed|false|
> |remaining|10243256320|
> |storage|DatanodeStorage  (id=6872)|



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10713) Throttle FsNameSystem lock warnings

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-10713:
-
  Resolution: Fixed
Hadoop Flags: Reviewed
   Fix Version/s: 3.0.0-alpha2
Target Version/s:   (was: 2.8.0)
  Status: Resolved  (was: Patch Available)

Thanks [~hanishakoneru] for the contribution and everyone else for the reviews 
and discussion. I pushed it to trunk with the following trivial edit to satisfy 
checkstyle.

{code}
-  && !longestReadLockHeldInterval.compareAndSet(localLongestReadLock, 
readLockInterval));
+  && !longestReadLockHeldInterval.compareAndSet(localLongestReadLock,
+readLockInterval));
{code}

Hanisha if you want to post a branch-2 patch I can commit that too, I think it 
just needs trivial changes to TestFSNameSystem.java to make variables accessed 
in the inner classes final since Java 7 won't allow that.

> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, 
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, 
> HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, 
> HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, 
> HDFS-10713.011.patch, HDFS-10713.012.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a 
> thread for over 1 second. These messages can be throttled to at one most one 
> per x minutes to avoid potentially filling up NN logs. We can also log the 
> number of suppressed notices since the last log message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10713) Throttle FsNameSystem lock warnings

[Arpit Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524583#comment-15524583
 ] 

Arpit Agarwal edited comment on HDFS-10713 at 9/27/16 12:17 AM:


Thanks [~hanishakoneru] for the contribution and everyone else for the reviews 
and discussion. I pushed it to trunk with the following trivial edit to satisfy 
checkstyle.

{code}
-  && !longestReadLockHeldInterval.compareAndSet(localLongestReadLock, 
readLockInterval));
+  && !longestReadLockHeldInterval.compareAndSet(localLongestReadLock,
+readLockInterval));
{code}

Hanisha if you want to post a branch-2 patch I can commit that too, I think it 
just needs trivial changes to TestFSNameSystem.java to make variables accessed 
in the inner classes final since Java 7 won't allow access to non-final vars.


was (Author: arpitagarwal):
Thanks [~hanishakoneru] for the contribution and everyone else for the reviews 
and discussion. I pushed it to trunk with the following trivial edit to satisfy 
checkstyle.

{code}
-  && !longestReadLockHeldInterval.compareAndSet(localLongestReadLock, 
readLockInterval));
+  && !longestReadLockHeldInterval.compareAndSet(localLongestReadLock,
+readLockInterval));
{code}

Hanisha if you want to post a branch-2 patch I can commit that too, I think it 
just needs trivial changes to TestFSNameSystem.java to make variables accessed 
in the inner classes final since Java 7 won't allow that.

> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, 
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, 
> HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, 
> HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, 
> HDFS-10713.011.patch, HDFS-10713.012.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a 
> thread for over 1 second. These messages can be throttled to at one most one 
> per x minutes to avoid potentially filling up NN logs. We can also log the 
> number of suppressed notices since the last log message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10897) Ozone: SCM: Add NodeManager

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524553#comment-15524553
 ] 

Hadoop QA commented on HDFS-10897:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  7m 
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m 
 5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
7s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
16s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 24s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  1m 
13s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 
unchanged - 0 fixed = 8 total (was 7) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 20s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 51s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestBlockPoolManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10897 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830408/HDFS-10897-HDFS-7240.003.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux e53fcf42efed 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | HDFS-7240 / 0f73cd7 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16869/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| javadoc | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16869/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16869/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16869/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16869/console |
| Powered by | Apache 

[jira] [Updated] (HDFS-10892) Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HDFS-10892:
-
Attachment: HDFS-10892.003.patch

> Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'
> ---
>
> Key: HDFS-10892
> URL: https://issues.apache.org/jira/browse/HDFS-10892
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10892.000.patch, HDFS-10892.001.patch, 
> HDFS-10892.002.patch, HDFS-10892.003.patch
>
>
> I did not find unit test in {{trunk}} code for following cases:
> - HDFS command {{dfs -tail}}
> - HDFS command {{dfs -stat}}
> - file name or content with UTF-8 characters
> I think it still merits to have one though the commands have served us for 
> years.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.

[Jing Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524408#comment-15524408
 ] 

Jing Zhao commented on HDFS-10826:
--

Thanks for working on this, [~tasanuma0829]. And thanks for the review, 
[~ajisakaa].

The patch looks good to me. One minor comment is about the change in 
NameNodeFsck: currently we do not print detailed replica info for a missing 
replicated block. However for a striped block with less than min-replication 
number of internal blocks, we may still want to print out its internal block 
information even though we can claim it as MISSING.

> The result of fsck should be CRITICAL when there are unrecoverable ec block 
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
>  Total size:536870912 B
>  Total files:   1
>  Total block groups (validated):1 (avg. block group size 536870912 B)
>   
>   UNRECOVERABLE BLOCK GROUPS:   1 (100.0 %)
>   
>  Minimally erasure-coded block groups:  0 (0.0 %)
>  Over-erasure-coded block groups:   0 (0.0 %)
>  Under-erasure-coded block groups:  1 (100.0 %)
>  Unsatisfactory placement block groups: 0 (0.0 %)
>  Default ecPolicy:  RS-DEFAULT-6-3-64k
>  Average block group size:  5.0
>  Missing block groups:  0
>  Corrupt block groups:  0
>  Missing internal blocks:   4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL. 
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is 
> unrecoverable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10609) Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524395#comment-15524395
 ] 

Hudson commented on HDFS-10609:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10490 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10490/])
Addendum patch for HDFS-10609. Uncaught InvalidEncryptionKeyException (weichiu: 
rev 8e06d865c4848f2ddd1a3ec4ee825e152d8e77c3)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptedTransfer.java


> Uncaught InvalidEncryptionKeyException during pipeline recovery may abort 
> downstream applications
> -
>
> Key: HDFS-10609
> URL: https://issues.apache.org/jira/browse/HDFS-10609
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: encryption
>Affects Versions: 2.6.0
> Environment: CDH5.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10609.001.patch, HDFS-10609.002.patch, 
> HDFS-10609.003.patch, HDFS-10609.004.patch, HDFS-10609.005.patch
>
>
> In normal operations, if SASL negotiation fails due to 
> {{InvalidEncryptionKeyException}}, it is typically a benign exception, which 
> is caught and retried :
> {code:title=SaslDataTransferServer#doSaslHandshake}
>   if (ioe instanceof SaslException &&
>   ioe.getCause() != null &&
>   ioe.getCause() instanceof InvalidEncryptionKeyException) {
> // This could just be because the client is long-lived and hasn't gotten
> // a new encryption key from the NN in a while. Upon receiving this
> // error, the client will get a new encryption key from the NN and retry
> // connecting to this DN.
> sendInvalidKeySaslErrorMessage(out, ioe.getCause().getMessage());
>   } 
> {code}
> {code:title=DFSOutputStream.DataStreamer#createBlockOutputStream}
> if (ie instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) {
> DFSClient.LOG.info("Will fetch a new encryption key and retry, " 
> + "encryption key was invalid when connecting to "
> + nodes[0] + " : " + ie);
> {code}
> However, if the exception is thrown during pipeline recovery, the 
> corresponding code does not handle it properly, and the exception is spilled 
> out to downstream applications, such as SOLR, aborting its operation:
> {quote}
> 2016-07-06 12:12:51,992 ERROR org.apache.solr.update.HdfsTransactionLog: 
> Exception closing tlog.
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: 
> Can't re-compute encryption key for nonce, since the required block key 
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:474)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:1308)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1272)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1433)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1147)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:632)
> 2016-07-06 12:12:51,997 ERROR org.apache.solr.update.CommitTracker: auto 
> commit error...:org.apache.solr.common.SolrException: 
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: 
> Can't re-compute encryption key for nonce, since the required block key 
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at 
> org.apache.solr.update.HdfsTransactionLog.close(HdfsTransactionLog.java:316)
> at 
> org.apache.solr.update.TransactionLog.decref(TransactionLog.java:505)
> at org.apache.solr.update.UpdateLog.addOldLog(UpdateLog.java:380)
> at 

[jira] [Updated] (HDFS-10637) Modifications to remove the assumption that FsVolumes are backed by java.io.File.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-10637:
--
Attachment: HDFS-10637.009.patch

Thanks [~eddyxu] for the clarification. I am posting a new patch to address 
your comments. I wanted to point out two in particular: 

* Re {{FsDatasetSpi#getFileInFinalizedDir()}} - I removed this. In the test 
cases where this was used (e.g. {{TestDirectoryScanner}}), I replaced the call 
with {{FsVolumeImpl#getFinalizedDir}}. {{FsVolumeImpl#getFinalizedDir}} is 
currently used only in the test cases, and is marked as such. Another way to 
address this (given the way {{FsVolumeImpl#getFinalizedDir}} is used in the 
test cases), could have been to add calls such as {{createBlock}} and 
{{createBlockMetadata}} to {{FsVolumeImpl}}, which create files for the block 
data and block metadata (respectively). The former re-uses existing functions, 
and seems cleaner. 

* Re  {{currentDir}} in {{FsVolumeImpl}} - I have not removed this yet. It is a 
private variable of {{FsVolumeImpl}}. I think having it makes the code simpler 
-- if not, any use of {{currentDir}} needs to be replaced by something like 
{{new File(StorageLocation.getFile(), STORAGE_DIR_CURRENT)}}. {{currentDir}} is 
now just a private variable that holds this value. What do you think?

> Modifications to remove the assumption that FsVolumes are backed by 
> java.io.File.
> -
>
> Key: HDFS-10637
> URL: https://issues.apache.org/jira/browse/HDFS-10637
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10637.001.patch, HDFS-10637.002.patch, 
> HDFS-10637.003.patch, HDFS-10637.004.patch, HDFS-10637.005.patch, 
> HDFS-10637.006.patch, HDFS-10637.007.patch, HDFS-10637.008.patch, 
> HDFS-10637.009.patch
>
>
> Modifications to {{FsVolumeSpi}} and {{FsVolumeImpl}} to remove references to 
> {{java.io.File}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Chen Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524363#comment-15524363
 ] 

Chen Liang commented on HDFS-10903:
---

Thanks [~xyao] and [~liuml07] for the comments!

It turns out there are quite a number of places with the same issue as 
{{io.file.buffer.size}}. So just as  [~liuml07] mentioned, we plan to have this 
JIRA for all such fixes in HDFS and HADOOP-13658 for all such fixes in 
hadoop-commons.

Will surely merge the change and include {{ContractTestUtils}} and update the 
patch.

> Replace config key literal strings with config key names II: hadoop hdfs
> 
>
> Key: HDFS-10903
> URL: https://issues.apache.org/jira/browse/HDFS-10903
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mingliang Liu
>Assignee: Chen Liang
>Priority: Minor
> Attachments: HADOOP-13644.001.patch
>
>
> In *Hadoop HDFS*, there are some places that use config key literal strings 
> instead of config key names, e.g.
> {code:title=IOUtils.java}
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
> {code}
> We should replace places like this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10314) A new tool to sync current HDFS view to specified snapshot

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524359#comment-15524359
 ] 

Yongjun Zhang commented on HDFS-10314:
--

Hi [~jingzhao],

To better address your comments, I discussed with [~atm], and we think there 
are the following possible options:

Option 1. To get closer to what you suggested, we can introduce new tool 
distsync and let it support all of the following semantics:
* distsync -diff s1 s2 src tgt
* distsync -rdiff s2 s1 src tgt
* distsync tgt s1

Implementation-wise, we can move common DistCp stuff to AbstractDistCp, and let 
both DistCp and DistSync inherit from AbstractDistCp.  Accordingly, 
DistCpOption need to be refactored in a similar fashion.

With this route, the "-diff" in distcp will probably eventually be obsoleted.

Option 2. Given that we already have -diff switch in DistCp, we can add 
"-rdiff" to DistCp, and support
* distcp -diff s1 s2 src tgt (currently supported)
* distcp -rdiff s2 s1 src tgt

Would you please share what you think?

Thanks a lot.








> A new tool to sync current HDFS view to specified snapshot
> --
>
> Key: HDFS-10314
> URL: https://issues.apache.org/jira/browse/HDFS-10314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-10314.001.patch
>
>
> HDFS-9820 proposed adding -rdiff switch to distcp, as a reversed operation of 
> -diff switch. 
> Upon discussion with [~jingzhao], we will introduce a new tool that wraps 
> around distcp to achieve the same purpose.
> I'm thinking about calling the new tool "rsync", similar to unix/linux 
> command "rsync". The "r" here means remote.
> The syntax that simulate -rdiff behavior proposed in HDFS-9820 is
> {code}
> rsync  
> {code}
> This command ensure   is newer than .
> I think, In the future, we can add another command to have the functionality 
> of -diff switch of distcp.
> {code}
> sync  
> {code}
> that ensures   is older than .
> Thanks [~jingzhao].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10887) Provide admin/debug tool to dump block map

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524347#comment-15524347
 ] 

Yongjun Zhang commented on HDFS-10887:
--

Many thanks [~kihwal]! Very helpful information!

{quote}
By simply dumping the blocksmap, you will find which blocks are being affected, 
but where do you go from there? How do you identify the datanodes that are not 
reporting?
{quote}
What I was thinking: we can issue a "find blk_..." command passwordlessly to 
all datanodes  and see which DNs have the block. And for DNs have the block, 
then check DN logs to see what the DNs are doing and why block report was not 
sent. 

Together with this info found, then check the dead DN list, it will give a good 
reason why we don't have block report for the block of interest. 

Do that make sense to you?

Thanks.







> Provide admin/debug tool to dump block map
> --
>
> Key: HDFS-10887
> URL: https://issues.apache.org/jira/browse/HDFS-10887
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: hdfs, namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-10887.001.patch, HDFS-10887.002.patch
>
>
> From time to time, when NN restarts, we see
> {code}
> "The reported blocks X needs additional Y blocks to reach the threshold 
> 0.9990 of total blocks Z. Safe mode will be turned off automatically.
> {code}
> We'd wonder what these blocks that still need block reports are, and what DNs 
> they could possibly be located, what happened to these DNs.
> This jira to to propose a new admin or debug tool to dump the block map info 
> with the blocks that have fewer than minRepl replicas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HDFS-10903:
-
Status: Open  (was: Patch Available)

> Replace config key literal strings with config key names II: hadoop hdfs
> 
>
> Key: HDFS-10903
> URL: https://issues.apache.org/jira/browse/HDFS-10903
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mingliang Liu
>Assignee: Chen Liang
>Priority: Minor
> Attachments: HADOOP-13644.001.patch
>
>
> In *Hadoop HDFS*, there are some places that use config key literal strings 
> instead of config key names, e.g.
> {code:title=IOUtils.java}
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
> {code}
> We should replace places like this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524331#comment-15524331
 ] 

Mingliang Liu commented on HDFS-10903:
--

Thanks [~xyao] for the review.

We plan to address the problem by modules for short/independent Jenkins run, 
and review. [HADOOP-13658] should cover the changes in hadoop common module, 
and this one is for hadoop HDFS module. [~vagarychen] can you merge the changes 
with [HADOOP-13658] and also address [~xyao]'s comments about the 
{{io.file.buffer.zie}} usage in {{o.a.h.fs.contract.ContractTestUtils.java}}?

> Replace config key literal strings with config key names II: hadoop hdfs
> 
>
> Key: HDFS-10903
> URL: https://issues.apache.org/jira/browse/HDFS-10903
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mingliang Liu
>Assignee: Chen Liang
>Priority: Minor
> Attachments: HADOOP-13644.001.patch
>
>
> In *Hadoop HDFS*, there are some places that use config key literal strings 
> instead of config key names, e.g.
> {code:title=IOUtils.java}
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
> {code}
> We should replace places like this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10897) Ozone: SCM: Add NodeManager

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-10897:

Attachment: HDFS-10897-HDFS-7240.003.patch

Addressed all comments from [~xyao] 

> Ozone: SCM: Add NodeManager
> ---
>
> Key: HDFS-10897
> URL: https://issues.apache.org/jira/browse/HDFS-10897
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Attachments: HDFS-10897-HDFS-7240.001.patch, 
> HDFS-10897-HDFS-7240.002.patch, HDFS-10897-HDFS-7240.003.patch
>
>
> Add a nodeManager class that will be used by Storage Controller Manager 
> eventually.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10905) Refactor DataStreamer#createBlockOutputStream

[Wei-Chiu Chuang (JIRA)]

Wei-Chiu Chuang created HDFS-10905:
--

 Summary: Refactor DataStreamer#createBlockOutputStream
 Key: HDFS-10905
 URL: https://issues.apache.org/jira/browse/HDFS-10905
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client
Affects Versions: 2.8.0
Reporter: Wei-Chiu Chuang
Priority: Minor


DataStreamer#createBlockOutputStream and DataStreamer#transfer shared much 
boilerplate code. HDFS-10609 refactored the transfer method into a 
StreamerStreams class. The createBlockOutputStream method should reuse the 
class to de-dup code and to improve code clarity.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Xiaoyu Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524298#comment-15524298
 ] 

Xiaoyu Yao edited comment on HDFS-10903 at 9/26/16 9:58 PM:


Thanks [~liuml07] for reporting this and [~vagarychen] for posting the patch. 
The change looks good to me. 
Can you also fix the "io.file.buffer.zie" usage in 
o.a.h.fs.contract.ContractTestUtils.java? +1 after that is fixed. 

Also, we should move this JIRA from Hadoop HDFS to Hadoop Common.


was (Author: xyao):
Thanks [~liuml07] for reporting this and [~vagarychen] for posting the patch. 
The change looks good to me. 
Can you also fix the "io.file.buffer.zie" usage in 
o.a.h.fs.contract.ContractTestUtils.java? +1 after that is fixed. 

> Replace config key literal strings with config key names II: hadoop hdfs
> 
>
> Key: HDFS-10903
> URL: https://issues.apache.org/jira/browse/HDFS-10903
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mingliang Liu
>Assignee: Chen Liang
>Priority: Minor
> Attachments: HADOOP-13644.001.patch
>
>
> In *Hadoop HDFS*, there are some places that use config key literal strings 
> instead of config key names, e.g.
> {code:title=IOUtils.java}
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
> {code}
> We should replace places like this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Xiaoyu Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524298#comment-15524298
 ] 

Xiaoyu Yao commented on HDFS-10903:
---

Thanks [~liuml07] for reporting this and [~vagarychen] for posting the patch. 
The change looks good to me. 
Can you also fix the "io.file.buffer.zie" usage in 
o.a.h.fs.contract.ContractTestUtils.java? +1 after that is fixed. 

> Replace config key literal strings with config key names II: hadoop hdfs
> 
>
> Key: HDFS-10903
> URL: https://issues.apache.org/jira/browse/HDFS-10903
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mingliang Liu
>Assignee: Chen Liang
>Priority: Minor
> Attachments: HADOOP-13644.001.patch
>
>
> In *Hadoop HDFS*, there are some places that use config key literal strings 
> instead of config key names, e.g.
> {code:title=IOUtils.java}
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
> {code}
> We should replace places like this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10609) Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-10609:
---
Release Note: If pipeline recovery fails due to expired encryption key, 
attempt to refresh the key and retry.

> Uncaught InvalidEncryptionKeyException during pipeline recovery may abort 
> downstream applications
> -
>
> Key: HDFS-10609
> URL: https://issues.apache.org/jira/browse/HDFS-10609
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: encryption
>Affects Versions: 2.6.0
> Environment: CDH5.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10609.001.patch, HDFS-10609.002.patch, 
> HDFS-10609.003.patch, HDFS-10609.004.patch, HDFS-10609.005.patch
>
>
> In normal operations, if SASL negotiation fails due to 
> {{InvalidEncryptionKeyException}}, it is typically a benign exception, which 
> is caught and retried :
> {code:title=SaslDataTransferServer#doSaslHandshake}
>   if (ioe instanceof SaslException &&
>   ioe.getCause() != null &&
>   ioe.getCause() instanceof InvalidEncryptionKeyException) {
> // This could just be because the client is long-lived and hasn't gotten
> // a new encryption key from the NN in a while. Upon receiving this
> // error, the client will get a new encryption key from the NN and retry
> // connecting to this DN.
> sendInvalidKeySaslErrorMessage(out, ioe.getCause().getMessage());
>   } 
> {code}
> {code:title=DFSOutputStream.DataStreamer#createBlockOutputStream}
> if (ie instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) {
> DFSClient.LOG.info("Will fetch a new encryption key and retry, " 
> + "encryption key was invalid when connecting to "
> + nodes[0] + " : " + ie);
> {code}
> However, if the exception is thrown during pipeline recovery, the 
> corresponding code does not handle it properly, and the exception is spilled 
> out to downstream applications, such as SOLR, aborting its operation:
> {quote}
> 2016-07-06 12:12:51,992 ERROR org.apache.solr.update.HdfsTransactionLog: 
> Exception closing tlog.
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: 
> Can't re-compute encryption key for nonce, since the required block key 
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:474)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:1308)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1272)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1433)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1147)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:632)
> 2016-07-06 12:12:51,997 ERROR org.apache.solr.update.CommitTracker: auto 
> commit error...:org.apache.solr.common.SolrException: 
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: 
> Can't re-compute encryption key for nonce, since the required block key 
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at 
> org.apache.solr.update.HdfsTransactionLog.close(HdfsTransactionLog.java:316)
> at 
> org.apache.solr.update.TransactionLog.decref(TransactionLog.java:505)
> at org.apache.solr.update.UpdateLog.addOldLog(UpdateLog.java:380)
> at org.apache.solr.update.UpdateLog.postCommit(UpdateLog.java:676)
> at 
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:623)
> at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> at 
> 

[jira] [Updated] (HDFS-10609) Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-10609:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
   2.8.0
   Status: Resolved  (was: Patch Available)

> Uncaught InvalidEncryptionKeyException during pipeline recovery may abort 
> downstream applications
> -
>
> Key: HDFS-10609
> URL: https://issues.apache.org/jira/browse/HDFS-10609
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: encryption
>Affects Versions: 2.6.0
> Environment: CDH5.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10609.001.patch, HDFS-10609.002.patch, 
> HDFS-10609.003.patch, HDFS-10609.004.patch, HDFS-10609.005.patch
>
>
> In normal operations, if SASL negotiation fails due to 
> {{InvalidEncryptionKeyException}}, it is typically a benign exception, which 
> is caught and retried :
> {code:title=SaslDataTransferServer#doSaslHandshake}
>   if (ioe instanceof SaslException &&
>   ioe.getCause() != null &&
>   ioe.getCause() instanceof InvalidEncryptionKeyException) {
> // This could just be because the client is long-lived and hasn't gotten
> // a new encryption key from the NN in a while. Upon receiving this
> // error, the client will get a new encryption key from the NN and retry
> // connecting to this DN.
> sendInvalidKeySaslErrorMessage(out, ioe.getCause().getMessage());
>   } 
> {code}
> {code:title=DFSOutputStream.DataStreamer#createBlockOutputStream}
> if (ie instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) {
> DFSClient.LOG.info("Will fetch a new encryption key and retry, " 
> + "encryption key was invalid when connecting to "
> + nodes[0] + " : " + ie);
> {code}
> However, if the exception is thrown during pipeline recovery, the 
> corresponding code does not handle it properly, and the exception is spilled 
> out to downstream applications, such as SOLR, aborting its operation:
> {quote}
> 2016-07-06 12:12:51,992 ERROR org.apache.solr.update.HdfsTransactionLog: 
> Exception closing tlog.
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: 
> Can't re-compute encryption key for nonce, since the required block key 
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:474)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:1308)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1272)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1433)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1147)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:632)
> 2016-07-06 12:12:51,997 ERROR org.apache.solr.update.CommitTracker: auto 
> commit error...:org.apache.solr.common.SolrException: 
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: 
> Can't re-compute encryption key for nonce, since the required block key 
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at 
> org.apache.solr.update.HdfsTransactionLog.close(HdfsTransactionLog.java:316)
> at 
> org.apache.solr.update.TransactionLog.decref(TransactionLog.java:505)
> at org.apache.solr.update.UpdateLog.addOldLog(UpdateLog.java:380)
> at org.apache.solr.update.UpdateLog.postCommit(UpdateLog.java:676)
> at 
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:623)
> at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> at 
> 

[jira] [Commented] (HDFS-10609) Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524276#comment-15524276
 ] 

Wei-Chiu Chuang commented on HDFS-10609:


Committed the last patch to trunk, branch-2 and branch-2.8. Thanks very much 
[~andrew.wang] [~xiaochen] for reviewing the patch!

> Uncaught InvalidEncryptionKeyException during pipeline recovery may abort 
> downstream applications
> -
>
> Key: HDFS-10609
> URL: https://issues.apache.org/jira/browse/HDFS-10609
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: encryption
>Affects Versions: 2.6.0
> Environment: CDH5.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HDFS-10609.001.patch, HDFS-10609.002.patch, 
> HDFS-10609.003.patch, HDFS-10609.004.patch, HDFS-10609.005.patch
>
>
> In normal operations, if SASL negotiation fails due to 
> {{InvalidEncryptionKeyException}}, it is typically a benign exception, which 
> is caught and retried :
> {code:title=SaslDataTransferServer#doSaslHandshake}
>   if (ioe instanceof SaslException &&
>   ioe.getCause() != null &&
>   ioe.getCause() instanceof InvalidEncryptionKeyException) {
> // This could just be because the client is long-lived and hasn't gotten
> // a new encryption key from the NN in a while. Upon receiving this
> // error, the client will get a new encryption key from the NN and retry
> // connecting to this DN.
> sendInvalidKeySaslErrorMessage(out, ioe.getCause().getMessage());
>   } 
> {code}
> {code:title=DFSOutputStream.DataStreamer#createBlockOutputStream}
> if (ie instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) {
> DFSClient.LOG.info("Will fetch a new encryption key and retry, " 
> + "encryption key was invalid when connecting to "
> + nodes[0] + " : " + ie);
> {code}
> However, if the exception is thrown during pipeline recovery, the 
> corresponding code does not handle it properly, and the exception is spilled 
> out to downstream applications, such as SOLR, aborting its operation:
> {quote}
> 2016-07-06 12:12:51,992 ERROR org.apache.solr.update.HdfsTransactionLog: 
> Exception closing tlog.
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: 
> Can't re-compute encryption key for nonce, since the required block key 
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:474)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:1308)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1272)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1433)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1147)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:632)
> 2016-07-06 12:12:51,997 ERROR org.apache.solr.update.CommitTracker: auto 
> commit error...:org.apache.solr.common.SolrException: 
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: 
> Can't re-compute encryption key for nonce, since the required block key 
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at 
> org.apache.solr.update.HdfsTransactionLog.close(HdfsTransactionLog.java:316)
> at 
> org.apache.solr.update.TransactionLog.decref(TransactionLog.java:505)
> at org.apache.solr.update.UpdateLog.addOldLog(UpdateLog.java:380)
> at org.apache.solr.update.UpdateLog.postCommit(UpdateLog.java:676)
> at 
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:623)
> at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> at 
> 

[jira] [Commented] (HDFS-10892) Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524272#comment-15524272
 ] 

Hadoop QA commented on HDFS-10892:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 25s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 2 new + 179 unchanged - 0 fixed = 181 total (was 179) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 50s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 27s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSShell |
|   | hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10892 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830377/HDFS-10892.002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 6b2f037bbd75 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 3ae652f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16867/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16867/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16867/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16867/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'
> ---
>
> Key: 

[jira] [Commented] (HDFS-10824) MiniDFSCluster#storageCapacities has no effects on real capacity

[Chris Nauroth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524258#comment-15524258
 ] 

Chris Nauroth commented on HDFS-10824:
--

[~xiaobingo], thank you for sharing patch 004.

Adding the {{waitDataNodeFullyStarted}} call for all DataNode restarts might 
slow down some tests that don't really need the wait.  Do you think that call 
can be moved inside {{setDataNodeStorageCapacities}}?  That way, you could do 
the wait only if {{storageCapacities}} is non-null and non-empty.

> MiniDFSCluster#storageCapacities has no effects on real capacity
> 
>
> Key: HDFS-10824
> URL: https://issues.apache.org/jira/browse/HDFS-10824
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10824.000.patch, HDFS-10824.001.patch, 
> HDFS-10824.002.patch, HDFS-10824.003.patch, HDFS-10824.004.patch
>
>
> It has been noticed MiniDFSCluster#storageCapacities has no effects on real 
> capacity. It can be reproduced by explicitly setting storageCapacities and 
> then call ClientProtocol#getDatanodeStorageReport(DatanodeReportType.LIVE) to 
> compare results. The following are  storage report for one node with two 
> volumes after I set capacity as 300 * 1024. Apparently, the capacity is not 
> changed.
> adminState|DatanodeInfo$AdminStates  (id=6861)
> |blockPoolUsed|215192|
> |cacheCapacity|0|
> |cacheUsed|0|
> |capacity|998164971520|
> |datanodeUuid|"839912e9-5bcb-45d1-81cf-9a9c9c02a00b" (id=6862)|
> |dependentHostNames|LinkedList  (id=6863)|
> |dfsUsed|215192|
> |hostName|"127.0.0.1" (id=6864)|
> |infoPort|64222|
> |infoSecurePort|0|
> |ipAddr|"127.0.0.1" (id=6865)|
> |ipcPort|64223|
> |lastUpdate|1472682790948|
> |lastUpdateMonotonic|209605640|
> |level|0|
> |location|"/default-rack" (id=6866)|
> |maintenanceExpireTimeInMS|0|
> |parent|null|
> |peerHostName|null|
> |remaining|20486512640|
> |softwareVersion|null|
> |upgradeDomain|null|
> |xceiverCount|1|
> |xferAddr|"127.0.0.1:64220" (id=6855)|
> |xferPort|64220|
> [0]StorageReport  (id=6856)
> |blockPoolUsed|4096|
> |capacity|499082485760|
> |dfsUsed|4096|
> |failed|false|
> |remaining|10243256320|
> |storage|DatanodeStorage  (id=6869)|
> [1]StorageReport  (id=6859)
> |blockPoolUsed|211096|
> |capacity|499082485760|
> |dfsUsed|211096|
> |failed|false|
> |remaining|10243256320|
> |storage|DatanodeStorage  (id=6872)|



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10609) Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524236#comment-15524236
 ] 

Hudson commented on HDFS-10609:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10489 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10489/])
HDFS-10609. Uncaught InvalidEncryptionKeyException during pipeline (weichiu: 
rev 3ae652f82110a52bf239f3c1849b48981558eb19)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockPoolTokenSecretManager.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptedTransfer.java


> Uncaught InvalidEncryptionKeyException during pipeline recovery may abort 
> downstream applications
> -
>
> Key: HDFS-10609
> URL: https://issues.apache.org/jira/browse/HDFS-10609
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: encryption
>Affects Versions: 2.6.0
> Environment: CDH5.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HDFS-10609.001.patch, HDFS-10609.002.patch, 
> HDFS-10609.003.patch, HDFS-10609.004.patch, HDFS-10609.005.patch
>
>
> In normal operations, if SASL negotiation fails due to 
> {{InvalidEncryptionKeyException}}, it is typically a benign exception, which 
> is caught and retried :
> {code:title=SaslDataTransferServer#doSaslHandshake}
>   if (ioe instanceof SaslException &&
>   ioe.getCause() != null &&
>   ioe.getCause() instanceof InvalidEncryptionKeyException) {
> // This could just be because the client is long-lived and hasn't gotten
> // a new encryption key from the NN in a while. Upon receiving this
> // error, the client will get a new encryption key from the NN and retry
> // connecting to this DN.
> sendInvalidKeySaslErrorMessage(out, ioe.getCause().getMessage());
>   } 
> {code}
> {code:title=DFSOutputStream.DataStreamer#createBlockOutputStream}
> if (ie instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) {
> DFSClient.LOG.info("Will fetch a new encryption key and retry, " 
> + "encryption key was invalid when connecting to "
> + nodes[0] + " : " + ie);
> {code}
> However, if the exception is thrown during pipeline recovery, the 
> corresponding code does not handle it properly, and the exception is spilled 
> out to downstream applications, such as SOLR, aborting its operation:
> {quote}
> 2016-07-06 12:12:51,992 ERROR org.apache.solr.update.HdfsTransactionLog: 
> Exception closing tlog.
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: 
> Can't re-compute encryption key for nonce, since the required block key 
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:474)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:1308)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1272)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1433)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1147)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:632)
> 2016-07-06 12:12:51,997 ERROR org.apache.solr.update.CommitTracker: auto 
> commit error...:org.apache.solr.common.SolrException: 
> 

[jira] [Updated] (HDFS-10896) Move lock logging logic from FSNamesystem into FSNamesystemLock

[Erik Krogen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Krogen updated HDFS-10896:
---
Attachment: HDFS-10896.001.patch

Rebased for patch v001.

> Move lock logging logic from FSNamesystem into FSNamesystemLock
> ---
>
> Key: HDFS-10896
> URL: https://issues.apache.org/jira/browse/HDFS-10896
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>  Labels: logging, namenode
> Attachments: HDFS-10896.000.patch, HDFS-10896.001.patch
>
>
> There are a number of tickets (HDFS-10742, HDFS-10817, HDFS-10713, this 
> subtask's story HDFS-10475) which are adding/improving logging/metrics around 
> the {{FSNamesystemLock}}. All of this is done in {{FSNamesystem}} right now, 
> which is polluting the namesystem with ThreadLocal variables, timing 
> counters, etc. which are only relevant to the lock itself and the number of 
> these increases as the logging/metrics become more sophisticated. It would be 
> best to move these all into FSNamesystemLock to keep the metrics/logging tied 
> directly to the item of interest. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9390) Block management for maintenance states

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524221#comment-15524221
 ] 

Hadoop QA commented on HDFS-9390:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 1057 unchanged - 9 fixed = 1058 total (was 1066) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m  7s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 44s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-9390 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830375/HDFS-9390-2.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux e1b11e4d348c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 4815d02 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16866/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16866/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16866/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16866/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> 

[jira] [Commented] (HDFS-10896) Move lock logging logic from FSNamesystem into FSNamesystemLock

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524223#comment-15524223
 ] 

Hadoop QA commented on HDFS-10896:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  5s{color} 
| {color:red} HDFS-10896 does not apply to trunk. Rebase required? Wrong 
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-10896 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830121/HDFS-10896.000.patch |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16868/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Move lock logging logic from FSNamesystem into FSNamesystemLock
> ---
>
> Key: HDFS-10896
> URL: https://issues.apache.org/jira/browse/HDFS-10896
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>  Labels: logging, namenode
> Attachments: HDFS-10896.000.patch
>
>
> There are a number of tickets (HDFS-10742, HDFS-10817, HDFS-10713, this 
> subtask's story HDFS-10475) which are adding/improving logging/metrics around 
> the {{FSNamesystemLock}}. All of this is done in {{FSNamesystem}} right now, 
> which is polluting the namesystem with ThreadLocal variables, timing 
> counters, etc. which are only relevant to the lock itself and the number of 
> these increases as the logging/metrics become more sophisticated. It would be 
> best to move these all into FSNamesystemLock to keep the metrics/logging tied 
> directly to the item of interest. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10900) DiskBalancer: Complete the documents for the report command

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524195#comment-15524195
 ] 

Anu Engineer commented on HDFS-10900:
-

I was about to commit this, and I noticed that hadoop documentation rendering 
is broken in Alpha-2. I will commit this, but after filing another JIRA. 

> DiskBalancer: Complete the documents for the report command
> ---
>
> Key: HDFS-10900
> URL: https://issues.apache.org/jira/browse/HDFS-10900
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10900.001.patch
>
>
> Now the documemnts of the command {{hdfs diskbalancer -report}} look not 
> completed. Two minors:
> * The usage of {{hdfs diskbalancer -report}} is missing in {{HDFSCommands.md}}
> * One subcommand  of report command {{hdfs diskbalancer -report -top}} is 
> missing in {{HDFSDiskBalancer.md}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10900) DiskBalancer: Complete the documents for the report command

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-10900:

Fix Version/s: (was: 2.9.0)
   3.0.0-alpha2

> DiskBalancer: Complete the documents for the report command
> ---
>
> Key: HDFS-10900
> URL: https://issues.apache.org/jira/browse/HDFS-10900
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10900.001.patch
>
>
> Now the documemnts of the command {{hdfs diskbalancer -report}} look not 
> completed. Two minors:
> * The usage of {{hdfs diskbalancer -report}} is missing in {{HDFSCommands.md}}
> * One subcommand  of report command {{hdfs diskbalancer -report -top}} is 
> missing in {{HDFSDiskBalancer.md}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice

[Jing Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524191#comment-15524191
 ] 

Jing Zhao commented on HDFS-10797:
--

Thanks for working on this, Sean. This is actually a known issue and the reason 
we currently choose to double the blocks is to make the semantic consistent 
with the scenario that the rename op moves the file out the original parent 
directory (e.g., into another snapshottable sub-tree or even a 
non-snapshottable subtree). In the later case, the moved files have to be 
counted in both the source directory and the target directory, since it belongs 
to snapshots of the source, and the current target directory.

The scenario described in this jira is a special case. I agree it's strange to 
find that a file is counted twice after renaming under the same directory. 
However, with the current change the semantic will become inconsistent. For 
e.g., we will sometimes count the file twice and sometimes only once in the 
following scenario:
1. move the file out of the original parent directory
2. optional: take a new snapshot
3. move the file back

Without step 2 then the file is counted only once, but with step 2 since the 
newly created item is in another snapshot diff the file will be counted twice. 
Or if the file is renamed into a subdirectory of its parent directory (/foo/bar 
--> /foo/subdir/bar), this file is still double counted. From the end user 
point of view this inconsistency is also strange.

So currently I'm leaning towards always doing double count. Thoughts?



> Disk usage summary of snapshots causes renamed blocks to get counted twice
> --
>
> Key: HDFS-10797
> URL: https://issues.apache.org/jira/browse/HDFS-10797
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch, 
> HDFS-10797.003.patch
>
>
> DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how 
> much disk usage is used by a snapshot by tallying up the files in the 
> snapshot that have since been deleted (that way it won't overlap with regular 
> files whose disk usage is computed separately). However that is determined 
> from a diff that shows moved (to Trash or otherwise) or renamed files as a 
> deletion and a creation operation that may overlap with the list of blocks. 
> Only the deletion operation is taken into consideration, and this causes 
> those blocks to get represented twice in the disk usage tallying.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8406) Lease recovery continually failed

[Noe (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524189#comment-15524189
 ] 

Noe commented on HDFS-8406:
---

for those that run into this, but need to get system working again:
1) move problem wal to temp directory
2) then cp the file back into original location



> Lease recovery continually failed
> -
>
> Key: HDFS-8406
> URL: https://issues.apache.org/jira/browse/HDFS-8406
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Keith Turner
>  Labels: Accumulo, HBase, SolrCloud
>
> While testing Accumulo on a cluster and killing processes, I ran into a 
> situation where the lease on an accumulo write ahead log in HDFS could not be 
> recovered.   Even restarting HDFS and Accumulo would not fix the problem.
> The following message was seen in an Accumulo tablet server log immediately 
> before the tablet server was killed.
> {noformat}
> 2015-05-14 17:12:37,466 [hdfs.DFSClient] WARN : DFSOutputStream 
> ResponseProcessor exception  for block 
> BP-802741494-10.1.5.6-1431557089849:blk_1073932823_192060
> java.io.IOException: Bad response ERROR for block 
> BP-802741494-10.1.5.6-1431557089849:blk_1073932823_192060 from datanode 
> 10.1.5.9:50010
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:897)
> 2015-05-14 17:12:37,466 [hdfs.DFSClient] WARN : Error Recovery for block 
> BP-802741494-10.1.5.6-1431557089849:blk_1073932823_192060 in pipeline 
> 10.1.5.55:50010, 10.1.5.9:5
> {noformat}
> Before recovering data from a write ahead log, the Accumulo master attempts 
> to recover the lease.   This repeatedly failed with messages like the 
> following.
> {noformat}
> 2015-05-14 17:14:54,301 [recovery.HadoopLogCloser] WARN : Error recovering 
> lease on 
> hdfs://10.1.5.6:1/accumulo/wal/worker11+9997/3a731759-3594-4535-8086-245eed7cd4c2
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException):
>  failed to create file 
> /accumulo/wal/worker11+9997/3a731759-3594-4535-8086-245eed7cd4c2 for 
> DFSClient_NONMAPREDUCE_950713214_16 for client 10.1.5.158 because 
> pendingCreates is non-null but no leases found.
> {noformat}
> Below is some info from the NN logs for the problematic file.
> {noformat}
> [ec2-user@leader2 logs]$ grep 3a731759-3594-4535-8086-245 
> hadoop-ec2-user-namenode-leader2.log 
> 2015-05-14 17:10:46,299 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* 
> allocateBlock: 
> /accumulo/wal/worker11+9997/3a731759-3594-4535-8086-245eed7cd4c2. 
> BP-802741494-10.1.5.6-1431557089849 
> blk_1073932823_192060{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, 
> replicas=[ReplicaUnderConstruction[[DISK]DS-ffe07d7d-0e68-45b8-b3d5-c976f1716481:NORMAL:10.1.5.55:50010|RBW],
>  
> ReplicaUnderConstruction[[DISK]DS-6efec702-3f1f-4ec0-a31f-de947e7e6097:NORMAL:10.1.5.9:50010|RBW],
>  
> ReplicaUnderConstruction[[DISK]DS-5e27df17-abf8-47df-b4bc-c38d0cd426ea:NORMAL:10.1.5.45:50010|RBW]]}
> 2015-05-14 17:10:46,628 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* 
> fsync: /accumulo/wal/worker11+9997/3a731759-3594-4535-8086-245eed7cd4c2 for 
> DFSClient_NONMAPREDUCE_-1128465883_16
> 2015-05-14 17:14:49,288 INFO 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: recoverLease: [Lease.  
> Holder: DFSClient_NONMAPREDUCE_-1128465883_16, pendingcreates: 1], 
> src=/accumulo/wal/worker11+9997/3a731759-3594-4535-8086-245eed7cd4c2 from 
> client DFSClient_NONMAPREDUCE_-1128465883_16
> 2015-05-14 17:14:49,288 INFO 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering [Lease.  
> Holder: DFSClient_NONMAPREDUCE_-1128465883_16, pendingcreates: 1], 
> src=/accumulo/wal/worker11+9997/3a731759-3594-4535-8086-245eed7cd4c2
> 2015-05-14 17:14:49,289 WARN org.apache.hadoop.hdfs.StateChange: DIR* 
> NameSystem.internalReleaseLease: File 
> /accumulo/wal/worker11+9997/3a731759-3594-4535-8086-245eed7cd4c2 has not been 
> closed. Lease recovery is in progress. RecoveryId = 192257 for block 
> blk_1073932823_192060{blockUCState=UNDER_RECOVERY, primaryNodeIndex=2, 
> replicas=[ReplicaUnderConstruction[[DISK]DS-ffe07d7d-0e68-45b8-b3d5-c976f1716481:NORMAL:10.1.5.55:50010|RBW],
>  
> ReplicaUnderConstruction[[DISK]DS-6efec702-3f1f-4ec0-a31f-de947e7e6097:NORMAL:10.1.5.9:50010|RBW],
>  
> ReplicaUnderConstruction[[DISK]DS-5e27df17-abf8-47df-b4bc-c38d0cd426ea:NORMAL:10.1.5.45:50010|RBW]]}
> java.lang.IllegalStateException: Failed to finalize INodeFile 
> 3a731759-3594-4535-8086-245eed7cd4c2 since blocks[0] is non-complete, where 
> blocks=[blk_1073932823_192257{blockUCState=COMMITTED, primaryNodeIndex=2, 
> replicas=[ReplicaUnderConstruction[[DISK]DS-ffe07d7d-0e68-45b8-b3d5-c976f1716481:NORMAL:10.1.5.55:50010|RBW],
>  
> 

[jira] [Commented] (HDFS-10897) Ozone: SCM: Add NodeManager

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524169#comment-15524169
 ] 

Anu Engineer commented on HDFS-10897:
-

[~xyao] Thanks for the comments, I will post an updated patch soon.

> Ozone: SCM: Add NodeManager
> ---
>
> Key: HDFS-10897
> URL: https://issues.apache.org/jira/browse/HDFS-10897
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Attachments: HDFS-10897-HDFS-7240.001.patch, 
> HDFS-10897-HDFS-7240.002.patch
>
>
> Add a nodeManager class that will be used by Storage Controller Manager 
> eventually.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10897) Ozone: SCM: Add NodeManager

[Xiaoyu Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524153#comment-15524153
 ] 

Xiaoyu Yao commented on HDFS-10897:
---

Thanks [~anu] for working on this and post the patch. I took a look at the 
patch and it looks good to me overall. 
Below are my early feedback on the code. I will post my comments on unit tests 
later.
1. NodeManager.java
  NIT: I see we have registerDataeNode/removeDataNode and 
getNodeCount/getAllNodes().
  Can we have a consistent name, e.g., all with/wo .  
  
2. SCMNodeManager.java
typo in comments:  stale informatio. 

Can we use the wrapper from HadoopExecutors?
{code}
  executorService = Executors.newScheduledThreadPool(1,
 131new ThreadFactoryBuilder().setDaemon(true)
 132.setNameFormat("SCM Heartbeat Processing Thread - 
%d").build());
{code}

updateHeartbeat() Can you add comments to the synchronization assumption from 
the caller if any
similar to registerDatanode()?

removeDatanode() : empty overwrite implementation. Either add a TODO: or remove
it if it is not needed.

getNodes() : declare "throws IllegalArgumentException"

> Ozone: SCM: Add NodeManager
> ---
>
> Key: HDFS-10897
> URL: https://issues.apache.org/jira/browse/HDFS-10897
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Attachments: HDFS-10897-HDFS-7240.001.patch, 
> HDFS-10897-HDFS-7240.002.patch
>
>
> Add a nodeManager class that will be used by Storage Controller Manager 
> eventually.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9482) Replace DatanodeInfo constructors with a builder pattern

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524104#comment-15524104
 ] 

Hadoop QA commented on HDFS-9482:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
27s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 8 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
19s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 40s{color} | {color:orange} hadoop-hdfs-project: The patch generated 36 new 
+ 817 unchanged - 16 fixed = 853 total (was 833) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
52s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 2 new 
+ 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
59s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 59m 
33s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 39s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
|  |  Unread field:DatanodeInfoBuilder.java:[line 51] |
|  |  Unread field:DatanodeInfoBuilder.java:[line 77] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-9482 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830366/HDFS-9482.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 757e27fe6ad2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 4815d02 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16864/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
 |
| findbugs | 

[jira] [Commented] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524103#comment-15524103
 ] 

Hadoop QA commented on HDFS-10903:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 32s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch 
generated 2 new + 453 unchanged - 3 fixed = 455 total (was 456) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m  
4s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m  3s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10903 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830109/HADOOP-13644.001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 494e21e4aa79 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 4815d02 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16865/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16865/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16865/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Replace config key literal strings with config key names II: hadoop hdfs
> 
>
> Key: HDFS-10903
> URL: https://issues.apache.org/jira/browse/HDFS-10903
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mingliang Liu
>

[jira] [Updated] (HDFS-10897) Ozone: SCM: Add NodeManager

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-10897:

Attachment: HDFS-10897-HDFS-7240.002.patch

Fix the timeout issue in the test.

> Ozone: SCM: Add NodeManager
> ---
>
> Key: HDFS-10897
> URL: https://issues.apache.org/jira/browse/HDFS-10897
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Attachments: HDFS-10897-HDFS-7240.001.patch, 
> HDFS-10897-HDFS-7240.002.patch
>
>
> Add a nodeManager class that will be used by Storage Controller Manager 
> eventually.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524006#comment-15524006
 ] 

Hadoop QA commented on HDFS-10903:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 29s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch 
generated 2 new + 453 unchanged - 3 fixed = 455 total (was 456) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 49s{color} 
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 40s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
| Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13644 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830109/HADOOP-13644.001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux d63f1af94d30 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 4815d02 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10594/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10594/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10594/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10594/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Replace config key literal strings with config key names II: hadoop hdfs
> 

[jira] [Updated] (HDFS-10896) Move lock logging logic from FSNamesystem into FSNamesystemLock

[Erik Krogen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Krogen updated HDFS-10896:
---
Status: Patch Available  (was: Open)

> Move lock logging logic from FSNamesystem into FSNamesystemLock
> ---
>
> Key: HDFS-10896
> URL: https://issues.apache.org/jira/browse/HDFS-10896
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>  Labels: logging, namenode
> Attachments: HDFS-10896.000.patch
>
>
> There are a number of tickets (HDFS-10742, HDFS-10817, HDFS-10713, this 
> subtask's story HDFS-10475) which are adding/improving logging/metrics around 
> the {{FSNamesystemLock}}. All of this is done in {{FSNamesystem}} right now, 
> which is polluting the namesystem with ThreadLocal variables, timing 
> counters, etc. which are only relevant to the lock itself and the number of 
> these increases as the logging/metrics become more sophisticated. It would be 
> best to move these all into FSNamesystemLock to keep the metrics/logging tied 
> directly to the item of interest. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10609) Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523899#comment-15523899
 ] 

Wei-Chiu Chuang commented on HDFS-10609:


Committing this based on Andrew's +1.

> Uncaught InvalidEncryptionKeyException during pipeline recovery may abort 
> downstream applications
> -
>
> Key: HDFS-10609
> URL: https://issues.apache.org/jira/browse/HDFS-10609
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: encryption
>Affects Versions: 2.6.0
> Environment: CDH5.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HDFS-10609.001.patch, HDFS-10609.002.patch, 
> HDFS-10609.003.patch, HDFS-10609.004.patch, HDFS-10609.005.patch
>
>
> In normal operations, if SASL negotiation fails due to 
> {{InvalidEncryptionKeyException}}, it is typically a benign exception, which 
> is caught and retried :
> {code:title=SaslDataTransferServer#doSaslHandshake}
>   if (ioe instanceof SaslException &&
>   ioe.getCause() != null &&
>   ioe.getCause() instanceof InvalidEncryptionKeyException) {
> // This could just be because the client is long-lived and hasn't gotten
> // a new encryption key from the NN in a while. Upon receiving this
> // error, the client will get a new encryption key from the NN and retry
> // connecting to this DN.
> sendInvalidKeySaslErrorMessage(out, ioe.getCause().getMessage());
>   } 
> {code}
> {code:title=DFSOutputStream.DataStreamer#createBlockOutputStream}
> if (ie instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) {
> DFSClient.LOG.info("Will fetch a new encryption key and retry, " 
> + "encryption key was invalid when connecting to "
> + nodes[0] + " : " + ie);
> {code}
> However, if the exception is thrown during pipeline recovery, the 
> corresponding code does not handle it properly, and the exception is spilled 
> out to downstream applications, such as SOLR, aborting its operation:
> {quote}
> 2016-07-06 12:12:51,992 ERROR org.apache.solr.update.HdfsTransactionLog: 
> Exception closing tlog.
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: 
> Can't re-compute encryption key for nonce, since the required block key 
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:474)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:1308)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1272)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1433)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1147)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:632)
> 2016-07-06 12:12:51,997 ERROR org.apache.solr.update.CommitTracker: auto 
> commit error...:org.apache.solr.common.SolrException: 
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: 
> Can't re-compute encryption key for nonce, since the required block key 
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at 
> org.apache.solr.update.HdfsTransactionLog.close(HdfsTransactionLog.java:316)
> at 
> org.apache.solr.update.TransactionLog.decref(TransactionLog.java:505)
> at org.apache.solr.update.UpdateLog.addOldLog(UpdateLog.java:380)
> at org.apache.solr.update.UpdateLog.postCommit(UpdateLog.java:676)
> at 
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:623)
> at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>

[jira] [Updated] (HDFS-10904) Need a new Result state for DiskBalancerWorkStatus to indicate the final Plan step errors and stuck rebalancing

[Manoj Govindassamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manoj Govindassamy updated HDFS-10904:
--
Description: 
* A DiskBalancer {{NodePlan}} might include a Single {{MoveStep}} or a list of 
MoveSteps to perform the requested disk balancing operation.

* {{DiskBalancerWorkStatus}} tracks the current disk balancing operation status 
for the {{Plan}} just submitted. 

* {{DiskBalancerWorkStatus#Result}} has following states and the state machine 
movement for the {{currentResult}} state doesn't seem to be a driven totally 
from disk balancing operation. Especially, the state movement to DONE is 
happening only upon QueryResult, which can be improved. {code}
  /** Various result values. **/
  public enum Result {
NO_PLAN(0),
PLAN_UNDER_PROGRESS(1),
PLAN_DONE(2),
PLAN_CANCELLED(3);

DiskBalancer
cancelPlan(String)
this.currentResult = Result.PLAN_CANCELLED;
DiskBalancer(String, Configuration, BlockMover)
this.currentResult = Result.NO_PLAN;
queryWorkStatus()
this.currentResult = Result.PLAN_DONE;
shutdown()
  this.currentResult = Result.NO_PLAN;
this.currentResult = Result.PLAN_CANCELLED;
submitPlan(String, long, String, String, boolean)
  this.currentResult = Result.PLAN_UNDER_PROGRESS;
{code}


* More importantly, when the final {{MoveStep}} of the {{NodePlan}} fails, the 
currentResult state is stuck in {{PLAN_UNDER_PROGRESS}} forever. User querying 
the status will assume the operation is in progress when in reality its not 
making any progress.  User can also run {{Query}} command with _verbose_ option 
which then will display more details about the operation which includes details 
about errors encountered.
**  Query Output: {code}
Plan File:  <_file_path_>
Plan ID: <_plan_hash_>
Result: PLAN_UNDER_PROGRESS
{code}

** {code}
"sourcePath" : "/data/disk2/hdfs/dn",
  "destPath" : "/data/disk3/hdfs/dn",
  "workItem" :
.. .. ..
"errorCount" : 0,
"errMsg" : null,
.. .. 
"maxDiskErrors" : 5,
.. .. ..
{code}
** But, user has to decipher these details to make out that the disk balancing 
operation is stuck as the top level Result still says {{PLAN_UNDER_PROGRESS}}. 
So, we want the DiskBalancer differentiate between the in-progress operation 
and the stuck or final error operations.


  was:
* A DiskBalancer {{NodePlan}} might include a Single {{MoveStep}} or a list of 
MoveSteps to perform the requested disk balancing operation.

* {{DiskBalancerWorkStatus}} tracks the current disk balancing operation status 
for the {{Plan}} just submitted. 

* {{DiskBalancerWorkStatus#Result}} has following states and the state machine 
movement for the {{currentResult}} state doesn't seem to be a driven totally 
from disk balancing operation. Especially, the state movement to DONE is 
happening only upon QueryResult, which can be improved.

{code}
  /** Various result values. **/
  public enum Result {
NO_PLAN(0),
PLAN_UNDER_PROGRESS(1),
PLAN_DONE(2),
PLAN_CANCELLED(3);

DiskBalancer
cancelPlan(String)
this.currentResult = Result.PLAN_CANCELLED;
DiskBalancer(String, Configuration, BlockMover)
this.currentResult = Result.NO_PLAN;
queryWorkStatus()
this.currentResult = Result.PLAN_DONE;
shutdown()
  this.currentResult = Result.NO_PLAN;
this.currentResult = Result.PLAN_CANCELLED;
submitPlan(String, long, String, String, boolean)
  this.currentResult = Result.PLAN_UNDER_PROGRESS;
{code}


* More importantly, when the final {{MoveStep}} of the {{NodePlan}} fails, the 
currentResult state is stuck in {{PLAN_UNDER_PROGRESS}} forever. User querying 
the status will assume the operation is in progress when in reality its not 
making any progress.  User can also run {{Query}} command with _verbose_ option 
which then will display more details about the operation which includes details 
about errors encountered.
**  Query Output: {code}
Plan File:  <_file_path_>
Plan ID: <_plan_hash_>
Result: PLAN_UNDER_PROGRESS
{code}

** {code}
"sourcePath" : "/data/disk2/hdfs/dn",
  "destPath" : "/data/disk3/hdfs/dn",
  "workItem" :
.. .. ..
"errorCount" : 0,
"errMsg" : null,
.. .. 
"maxDiskErrors" : 5,
.. .. ..
{code}
** But, user has to decipher these details to make out that the disk balancing 
operation is stuck as the top level Result still says {{PLAN_UNDER_PROGRESS}}. 
So, we want the DiskBalancer differentiate between the in-progress operation 
and the stuck or final error operations.



> Need a new Result state for DiskBalancerWorkStatus to indicate the final Plan 
> step errors and stuck rebalancing
> ---
>
> Key: HDFS-10904
> URL: https://issues.apache.org/jira/browse/HDFS-10904
> Project: Hadoop HDFS
>  Issue Type: 

[jira] [Created] (HDFS-10904) Need a new Result state for DiskBalancerWorkStatus to indicate the final Plan step errors and stuck rebalancing

[Manoj Govindassamy (JIRA)]

Manoj Govindassamy created HDFS-10904:
-

 Summary: Need a new Result state for DiskBalancerWorkStatus to 
indicate the final Plan step errors and stuck rebalancing
 Key: HDFS-10904
 URL: https://issues.apache.org/jira/browse/HDFS-10904
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: balancer & mover
Affects Versions: 3.0.0-alpha2
Reporter: Manoj Govindassamy
Assignee: Manoj Govindassamy


* A DiskBalancer {{NodePlan}} might include a Single {{MoveStep}} or a list of 
MoveSteps to perform the requested disk balancing operation.

* {{DiskBalancerWorkStatus}} tracks the current disk balancing operation status 
for the {{Plan}} just submitted. 

* {{DiskBalancerWorkStatus#Result}} has following states and the state machine 
movement for the {{currentResult}} state doesn't seem to be a driven totally 
from disk balancing operation. Especially, the state movement to DONE is 
happening only upon QueryResult, which can be improved.

{code}
  /** Various result values. **/
  public enum Result {
NO_PLAN(0),
PLAN_UNDER_PROGRESS(1),
PLAN_DONE(2),
PLAN_CANCELLED(3);

DiskBalancer
cancelPlan(String)
this.currentResult = Result.PLAN_CANCELLED;
DiskBalancer(String, Configuration, BlockMover)
this.currentResult = Result.NO_PLAN;
queryWorkStatus()
this.currentResult = Result.PLAN_DONE;
shutdown()
  this.currentResult = Result.NO_PLAN;
this.currentResult = Result.PLAN_CANCELLED;
submitPlan(String, long, String, String, boolean)
  this.currentResult = Result.PLAN_UNDER_PROGRESS;
{code}


* More importantly, when the final {{MoveStep}} of the {{NodePlan}} fails, the 
currentResult state is stuck in {{PLAN_UNDER_PROGRESS}} forever. User querying 
the status will assume the operation is in progress when in reality its not 
making any progress.  User can also run {{Query}} command with _verbose_ option 
which then will display more details about the operation which includes details 
about errors encountered.
**  Query Output: {code}
Plan File:  <_file_path_>
Plan ID: <_plan_hash_>
Result: PLAN_UNDER_PROGRESS
{code}

** {code}
"sourcePath" : "/data/disk2/hdfs/dn",
  "destPath" : "/data/disk3/hdfs/dn",
  "workItem" :
.. .. ..
"errorCount" : 0,
"errMsg" : null,
.. .. 
"maxDiskErrors" : 5,
.. .. ..
{code}
** But, user has to decipher these details to make out that the disk balancing 
operation is stuck as the top level Result still says {{PLAN_UNDER_PROGRESS}}. 
So, we want the DiskBalancer differentiate between the in-progress operation 
and the stuck or final error operations.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9885) In distcp cmd ouput, Display name should be given for org.apache.hadoop.tools.mapred.CopyMapper$Counter.

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523789#comment-15523789
 ] 

Yongjun Zhang commented on HDFS-9885:
-

Thanks for pinging me [~brahmareddy], I took a look and the change looks good 
to me. Very sorry [~surendrasingh] for having missed your ping earlier.



> In distcp cmd ouput, Display name should be given for 
> org.apache.hadoop.tools.mapred.CopyMapper$Counter.
> 
>
> Key: HDFS-9885
> URL: https://issues.apache.org/jira/browse/HDFS-9885
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: distcp
>Affects Versions: 2.7.1
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Attachments: HDFS-9885.001.patch, HDFS-9885.002.patch
>
>
> In distcp cmd output,
> hadoop distcp hdfs://NN1:port/file1 hdfs://NN2:port/file2
> 16/02/29 07:05:55 INFO tools.DistCp: DistCp job-id: job_1456729398560_0002
> 16/02/29 07:05:55 INFO mapreduce.Job: Running job: job_1456729398560_0002
> 16/02/29 07:06:01 INFO mapreduce.Job: Job job_1456729398560_0002 running in 
> uber mode : false
> 16/02/29 07:06:01 INFO mapreduce.Job: map 0% reduce 0%
> 16/02/29 07:06:06 INFO mapreduce.Job: map 100% reduce 0%
> 16/02/29 07:06:07 INFO mapreduce.Job: Job job_1456729398560_0002 completed 
> successfully
> ...
> ...
> File Input Format Counters
> Bytes Read=212
> File Output Format Counters
> Bytes Written=0{color:red} 
> org.apache.hadoop.tools.mapred.CopyMapper$Counter
> {color}
> BANDWIDTH_IN_BYTES=12418
> BYTESCOPIED=12418
> BYTESEXPECTED=12418
> COPY=1
> Expected:
> Display Name can be given instead of 
> {color:red}"org.apache.hadoop.tools.mapred.CopyMapper$Counter"{color}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10892) Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HDFS-10892:
-
Description: 
I did not find unit test in {{trunk}} code for following cases:
- HDFS command {{dfs -tail}}
- HDFS command {{dfs -stat}}
- file name or content with UTF-8 characters

I think it still merits to have one though the commands have served us for 
years.

  was:I did not find unit test for HDFS command 'dfs -tail' in {{trunk}} code. 
I think it still merits to have one though the command itself has been used for 
years.


> Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'
> ---
>
> Key: HDFS-10892
> URL: https://issues.apache.org/jira/browse/HDFS-10892
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10892.000.patch, HDFS-10892.001.patch, 
> HDFS-10892.002.patch
>
>
> I did not find unit test in {{trunk}} code for following cases:
> - HDFS command {{dfs -tail}}
> - HDFS command {{dfs -stat}}
> - file name or content with UTF-8 characters
> I think it still merits to have one though the commands have served us for 
> years.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HDFS-10903:
-
Description: 
In *Hadoop HDFS*, there are some places that use config key literal strings 
instead of config key names, e.g.
{code:title=IOUtils.java}
copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
{code}

We should replace places like this.

  was:
There are some places that use config key literal strings instead of config key 
names, e.g.
{code:title=IOUtils.java}
copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
{code}

We should replace places like this.


> Replace config key literal strings with config key names II: hadoop hdfs
> 
>
> Key: HDFS-10903
> URL: https://issues.apache.org/jira/browse/HDFS-10903
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mingliang Liu
>Assignee: Chen Liang
>Priority: Minor
> Attachments: HADOOP-13644.001.patch
>
>
> In *Hadoop HDFS*, there are some places that use config key literal strings 
> instead of config key names, e.g.
> {code:title=IOUtils.java}
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
> {code}
> We should replace places like this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Moved] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu moved HADOOP-13644 to HDFS-10903:
---

Target Version/s: 2.8.0  (was: 2.8.0)
 Component/s: (was: conf)
 Key: HDFS-10903  (was: HADOOP-13644)
 Project: Hadoop HDFS  (was: Hadoop Common)

> Replace config key literal strings with config key names II: hadoop hdfs
> 
>
> Key: HDFS-10903
> URL: https://issues.apache.org/jira/browse/HDFS-10903
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mingliang Liu
>Assignee: Chen Liang
>Priority: Minor
> Attachments: HADOOP-13644.001.patch
>
>
> There are some places that use config key literal strings instead of config 
> key names, e.g.
> {code:title=IOUtils.java}
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
> {code}
> We should replace places like this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9850) DiskBalancer : Explore removing references to FsVolumeSpi

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523771#comment-15523771
 ] 

Anu Engineer commented on HDFS-9850:


[~manojg] Thanks for updating the patch as well providing a fix for this 
issues. Couple of minor comments.

In {{DiskBalancer.java}}

Changes in {{queryWorkStatus}} can be pulled into a function and invoked twice 
for source and destination.

{{createWorkPlan}} -- We left the "Disk Balancer" -- in the datanode side so it 
is easy to grep for error messages in datanode logs. If you don't mind can you 
please put it back.

{{createWorkPlan}} -- In this error messages it is easier for people to puzzle 
out the volume from the path than from UUID. I think we should leave the path 
in the error message. Please scan the file for removal of "Disk Balancer" from 
logging, also add that string in LOG messages that are added new.

nit : {{Line : 655}} function comments say we are returning volume, but in 
reality we are returning volume UUID.

{{copyBlocks#1013, 1021}} : this.setExitFlag is spurious, since we return from 
the function in the next line. That flag never gets evaluated. Also would you 
be able to set the path of volume in the error string instead of the UUID of 
the volumes.



{{FsDataSetImpl.java#getFsVolumeReference}} Do we really need to a function to 
FsDataSet Interface ? Cannot we do this via getVolumeReferences and using a 
helper function in DiskBalancer.java itself, I don't think we should add this 
to FsDatasetSpi.


{{TestDiskBalancer.java#577}} -- You might want to assert that test failed 
along with logging an info for the reconfig error.

{{TestDiskBalancer.java#597}} -- We verify that we are getting 
DiskbalancerException, but not the payload inside the exception. Would it make 
sense to verify that error is indeed some kind of volume error and string 
verify that verifies whether it is the source or dest ? 

> DiskBalancer : Explore removing references to FsVolumeSpi 
> --
>
> Key: HDFS-9850
> URL: https://issues.apache.org/jira/browse/HDFS-9850
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Affects Versions: 3.0.0-alpha2
>Reporter: Anu Engineer
>Assignee: Manoj Govindassamy
> Attachments: HDFS-9850.001.patch, HDFS-9850.002.patch
>
>
> In HDFS-9671, [~arpitagarwal] commented that we should explore the 
> possibility of removing references to FsVolumeSpi at any point and only deal 
> with storage ID. We are not sure if this is possible, this JIRA is to explore 
> if that can be done without issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Xiaoyu Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523763#comment-15523763
 ] 

Xiaoyu Yao commented on HDFS-10690:
---

[~fenghua_hu], thanks for updating the patch. The patch v05 looks good to me 
overall. Nice catch for the NoSuchElementException instead null returned from 
LinkedMap#firstKey(). Two minor issues: 
1. Can we wrap this NoSuchElementException handling below into a helper 
function that returns null when NoSuchElementException is thrown from 
firstKey() call? 

2. Do we need {{if (eldestKey == null)}} after the try/catch? I think it is 
needed only if we wrap the try/catch with a helper function, which returns null 
when the list is empty.

{code}
Object eldestKey;
897 try {
898   eldestKey = evictableMmapped.firstKey();
899 } catch (NoSuchElementException e) {
900   break;
901 }
902 if (eldestKey == null) {
903   break;
904 }
{code}

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> HDFS-10690.003.patch, HDFS-10690.004.patch, HDFS-10690.005.patch, 
> ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10892) Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HDFS-10892:
-
Attachment: HDFS-10892.002.patch

> Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'
> ---
>
> Key: HDFS-10892
> URL: https://issues.apache.org/jira/browse/HDFS-10892
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10892.000.patch, HDFS-10892.001.patch, 
> HDFS-10892.002.patch
>
>
> I did not find unit test for HDFS command 'dfs -tail' in {{trunk}} code. I 
> think it still merits to have one though the command itself has been used for 
> years.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9444) Add utility to find set of available ephemeral ports to ServerSocketUtil

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HDFS-9444:
---
Affects Version/s: test

> Add utility to find set of available ephemeral ports to ServerSocketUtil
> 
>
> Key: HDFS-9444
> URL: https://issues.apache.org/jira/browse/HDFS-9444
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: test
>Reporter: Brahma Reddy Battula
>Assignee: Masatake Iwasaki
> Attachments: HDFS-9444.001.patch, HDFS-9444.002.patch, 
> HDFS-9444.003.patch, HDFS-9444.004.patch, HDFS-9444.005.patch, 
> HDFS-9444.006.patch
>
>
> Unit tests using MiniDFSCluster with namanode-ha enabled need set of port 
> numbers in advance. Because namenodes talk to each other, we can not set ipc 
> port to 0 in configuration to make namenodes decide port number on its own. 
> ServerSocketUtil should provide utility to find set of available ephemeral 
> port numbers for this.
> For example, TestEditLogTailer could fail due to {{java.net.BindException: 
> Address already in use}}.
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/2556/testReport/
> {noformat}
> java.net.BindException: Problem binding to [localhost:42477] 
> java.net.BindException: Address already in use; For more details see:  
> http://wiki.apache.org/hadoop/BindException
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:444)
>   at sun.nio.ch.Net.bind(Net.java:436)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at org.apache.hadoop.ipc.Server.bind(Server.java:469)
>   at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
>   at org.apache.hadoop.ipc.Server.(Server.java:2464)
>   at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
>   at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRolls(TestEditLogTailer.java:139)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testNN1TriggersLogRolls(TestEditLogTailer.java:114)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10851) FSDirStatAndListingOp: stop passing path as string

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523757#comment-15523757
 ] 

Hadoop QA commented on HDFS-10851:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 26s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 2 new + 178 unchanged - 16 fixed = 180 total (was 194) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m  7s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 53s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestMaintenanceState |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery 
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10851 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830354/HDFS-10851.1.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 964bf56b6f56 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 4815d02 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16863/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16863/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16863/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16863/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> FSDirStatAndListingOp: stop passing path as string
> --

[jira] [Commented] (HDFS-8312) Trash does not descent into child directories to check for permissions

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523753#comment-15523753
 ] 

Daryn Sharp commented on HDFS-8312:
---

Encountered a minor conflict with this patch so I took a look.  If you view 
this as a security hole, then adding a new flag to rename and expecting well 
behaved clients to pass the flag is meaningless.  I'll just use the original 
rename to toss your directory in the trash.

NN-side trash support is a terrible idea.  The NN should not be deleting 
directories as the super-user on behalf of non-priviledged users.  The general 
solution to support trash policies using various directories isn't 
straightforward.

> Trash does not descent into child directories to check for permissions
> --
>
> Key: HDFS-8312
> URL: https://issues.apache.org/jira/browse/HDFS-8312
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs, security
>Affects Versions: 2.2.0, 2.6.0, 2.7.2
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: HDFS-8312-001.patch, HDFS-8312-002.patch, 
> HDFS-8312-003.patch, HDFS-8312-004.patch, HDFS-8312-005.patch, 
> HDFS-8312-testcase.patch
>
>
> HDFS trash does not descent into child directory to check if user has 
> permission to delete files.  For example:
> Run the following command to initialize directory structure as super user:
> {code}
> hadoop fs -mkdir /BSS/level1
> hadoop fs -mkdir /BSS/level1/level2
> hadoop fs -mkdir /BSS/level1/level2/level3
> hadoop fs -put /tmp/appConfig.json /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chown user1:users /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chown -R user1:users /BSS/level1
> hadoop fs -chown -R 750 /BSS/level1
> hadoop fs -chmod -R 640 /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chmod 775 /BSS
> {code}
> Change to a normal user called user2. 
> When trash is enabled:
> {code}
> sudo su user2 -
> hadoop fs -rm -r /BSS/level1
> 15/05/01 16:51:20 INFO fs.TrashPolicyDefault: Namenode trash configuration: 
> Deletion interval = 3600 minutes, Emptier interval = 0 minutes.
> Moved: 'hdfs://bdvs323.svl.ibm.com:9000/BSS/level1' to trash at: 
> hdfs://bdvs323.svl.ibm.com:9000/user/user2/.Trash/Current
> {code}
> When trash is disabled:
> {code}
> /opt/ibm/biginsights/IHC/bin/hadoop fs -Dfs.trash.interval=0 -rm -r 
> /BSS/level1
> 15/05/01 16:58:31 INFO fs.TrashPolicyDefault: Namenode trash configuration: 
> Deletion interval = 0 minutes, Emptier interval = 0 minutes.
> rm: Permission denied: user=user2, access=ALL, 
> inode="/BSS/level1":user1:users:drwxr-x---
> {code}
> There is inconsistency between trash behavior and delete behavior.  When 
> trash is enabled, files owned by user1 is deleted by user2.  It looks like 
> trash does not recursively validate if the child directory files can be 
> removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9390) Block management for maintenance states

[Ming Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ming Ma updated HDFS-9390:
--
Attachment: HDFS-9390-2.patch

Updated patch to address the checkstyle and test code issues.

> Block management for maintenance states
> ---
>
> Key: HDFS-9390
> URL: https://issues.apache.org/jira/browse/HDFS-9390
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9390-2.patch, HDFS-9390.patch
>
>
> When a node is transitioned to/stay in/transitioned out of maintenance state, 
> we need to make sure blocks w.r.t. that nodes are properly handled.
> * When nodes are put into maintenance, it will first go to 
> ENTERING_MAINTENANCE, and make sure blocks are minimally replicated before 
> the nodes are transitioned to IN_MAINTENANCE.
> * Do not replica blocks when nodes are in maintenance states. Maintenance 
> replica will remain in BlockMaps and thus is still considered valid from 
> block replication point of view. In other words, putting a node to 
> “maintenance” mode won’t trigger BlockManager to replicate its blocks.
> * Do not invalidate replicas on node under maintenance. After any file's 
> replication factor is reduced, NN needs to invalidate some replicas. It 
> should exclude nodes under maintenance in the handling.
> * Do not put IN_MAINTENANCE replicas in LocatedBlock for read operation.
> * Do not allocate any new block on nodes under maintenance.
> * Have Balancer exclude nodes under maintenance.
> * Exclude nodes under maintenance for DN cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9444) Add utility to find set of available ephemeral ports to ServerSocketUtil

[Brahma Reddy Battula (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523745#comment-15523745
 ] 

Brahma Reddy Battula commented on HDFS-9444:


Ok.. will commit..

> Add utility to find set of available ephemeral ports to ServerSocketUtil
> 
>
> Key: HDFS-9444
> URL: https://issues.apache.org/jira/browse/HDFS-9444
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Masatake Iwasaki
> Attachments: HDFS-9444.001.patch, HDFS-9444.002.patch, 
> HDFS-9444.003.patch, HDFS-9444.004.patch, HDFS-9444.005.patch, 
> HDFS-9444.006.patch
>
>
> Unit tests using MiniDFSCluster with namanode-ha enabled need set of port 
> numbers in advance. Because namenodes talk to each other, we can not set ipc 
> port to 0 in configuration to make namenodes decide port number on its own. 
> ServerSocketUtil should provide utility to find set of available ephemeral 
> port numbers for this.
> For example, TestEditLogTailer could fail due to {{java.net.BindException: 
> Address already in use}}.
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/2556/testReport/
> {noformat}
> java.net.BindException: Problem binding to [localhost:42477] 
> java.net.BindException: Address already in use; For more details see:  
> http://wiki.apache.org/hadoop/BindException
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:444)
>   at sun.nio.ch.Net.bind(Net.java:436)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at org.apache.hadoop.ipc.Server.bind(Server.java:469)
>   at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
>   at org.apache.hadoop.ipc.Server.(Server.java:2464)
>   at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
>   at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRolls(TestEditLogTailer.java:139)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testNN1TriggersLogRolls(TestEditLogTailer.java:114)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9482) Replace DatanodeInfo constructors with a builder pattern

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HDFS-9482:
---
Status: Patch Available  (was: Open)

 uploaded the patch..[~arpitagarwal] can you plrease review..?

> Replace DatanodeInfo constructors with a builder pattern
> 
>
> Key: HDFS-9482
> URL: https://issues.apache.org/jira/browse/HDFS-9482
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9482.patch
>
>
> As per  [~arpitagarwal] comment 
> [here|https://issues.apache.org/jira/browse/HDFS-9038?focusedCommentId=15018761=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15018761],Replace
>  DatanodeInfo constructors with a builder pattern, 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9482) Replace DatanodeInfo constructors with a builder pattern

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HDFS-9482:
---
Description: As per  [~arpitagarwal] comment 
[here|https://issues.apache.org/jira/browse/HDFS-9038?focusedCommentId=15018761=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15018761],Replace
 DatanodeInfo constructors with a builder pattern,   (was: From [~arpitagarwal] 
comment 
[here|https://issues.apache.org/jira/browse/HDFS-9038?focusedCommentId=15018761=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15018761].)

> Replace DatanodeInfo constructors with a builder pattern
> 
>
> Key: HDFS-9482
> URL: https://issues.apache.org/jira/browse/HDFS-9482
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9482.patch
>
>
> As per  [~arpitagarwal] comment 
> [here|https://issues.apache.org/jira/browse/HDFS-9038?focusedCommentId=15018761=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15018761],Replace
>  DatanodeInfo constructors with a builder pattern, 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9482) Replace DatanodeInfo constructors with a builder pattern

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HDFS-9482:
---
Description: From [~arpitagarwal] comment 
[here|https://issues.apache.org/jira/browse/HDFS-9038?focusedCommentId=15018761=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15018761].

> Replace DatanodeInfo constructors with a builder pattern
> 
>
> Key: HDFS-9482
> URL: https://issues.apache.org/jira/browse/HDFS-9482
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9482.patch
>
>
> From [~arpitagarwal] comment 
> [here|https://issues.apache.org/jira/browse/HDFS-9038?focusedCommentId=15018761=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15018761].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9482) Replace DatanodeInfo constructors with a builder pattern

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HDFS-9482:
---
Attachment: HDFS-9482.patch

> Replace DatanodeInfo constructors with a builder pattern
> 
>
> Key: HDFS-9482
> URL: https://issues.apache.org/jira/browse/HDFS-9482
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9482.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org