[jira] [Commented] (HDFS-10893) Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test
[ https://issues.apache.org/jira/browse/HDFS-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535145#comment-15535145 ] Mingliang Liu commented on HDFS-10893: -- [~jnp] would you like review this? Thanks. > Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test > > > Key: HDFS-10893 > URL: https://issues.apache.org/jira/browse/HDFS-10893 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Mingliang Liu >Assignee: Mingliang Liu > Attachments: HDFS-10893-branch-2.000.patch, HDFS-10893.000.patch > > > It seems that setting up MiniDFSCluser once for all commands test will reduce > the total time. To share a global cluster, the tests should use individual > test directories to avoid conflict between test cases. Meanwhile, the > MiniDFSCluster should not use the default root data directory; or else tests > are not able to launch another cluster(s) by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535092#comment-15535092
]
Hadoop QA commented on HDFS-10826:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 514 unchanged - 1 fixed = 515 total (was 515) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 62m
21s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10826 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831055/HDFS-10826.5.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ca7ba174e8aa 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 10be459 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16942/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16942/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16942/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>
[jira] [Commented] (HDFS-10909) De-duplicate code in ErasureCodingWorker#initializeStripedReadThreadPool and DFSClient#initThreadsNumForStripedReads
[
https://issues.apache.org/jira/browse/HDFS-10909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535029#comment-15535029
]
Hadoop QA commented on HDFS-10909:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 70
unchanged - 1 fixed = 70 total (was 71) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
0s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10909 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831042/HDFS-10909.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux dcbe940e7c60 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 10be459 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16941/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16941/testReport/ |
| modules | C: hadoop-hdfs-pr
[jira] [Commented] (HDFS-10850) getEZForPath should NOT throw FNF
[
https://issues.apache.org/jira/browse/HDFS-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535022#comment-15535022
]
Rakesh R commented on HDFS-10850:
-
Thank you [~andrew.wang] for your time and discussions to get the consensus.
The patch looks good to me, apart from one minor checkstyle warning.
{code}Unused import - java.io.FileNotFoundException.{code}
+1 (non-binding)
> getEZForPath should NOT throw FNF
> -
>
> Key: HDFS-10850
> URL: https://issues.apache.org/jira/browse/HDFS-10850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.0
>Reporter: Daryn Sharp
>Assignee: Andrew Wang
>Priority: Blocker
> Attachments: HDSF-10850.001.patch
>
>
> HDFS-9433 made an incompatible change to the semantics of getEZForPath. It
> used to return the EZ of the closest ancestor path. It never threw FNF. A
> common use of getEZForPath to determining if a file can be renamed, or must
> be copied due to mismatched EZs. Notably, this has broken hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10908) Improve StripedBlockReader#createBlockReader error logging
[
https://issues.apache.org/jira/browse/HDFS-10908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535018#comment-15535018
]
Hadoop QA commented on HDFS-10908:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 72m
49s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10908 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831040/HDFS-10908.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c662bfce42b7 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 10be459 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16939/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16939/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve StripedBlockReader#createBlockReader error logging
> --
>
> Key: HDFS-10908
> URL: https://issues.apache.org/jira/browse/HDFS-10908
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Minor
> Attachments:
[jira] [Assigned] (HDFS-9675) BlockSender incorrectly output error logs with non-English locale
[
https://issues.apache.org/jira/browse/HDFS-9675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jagadesh Kiran N reassigned HDFS-9675:
--
Assignee: Jagadesh Kiran N
> BlockSender incorrectly output error logs with non-English locale
> -
>
> Key: HDFS-9675
> URL: https://issues.apache.org/jira/browse/HDFS-9675
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Akira Ajisaka
>Assignee: Jagadesh Kiran N
> Labels: i18n
>
> If the locale is non-English, the following code does not work.
> {code:title=BlockSender.java}
> String ioem = e.getMessage();
> if (!ioem.startsWith("Broken pipe") && !ioem.startsWith("Connection
> reset")) {
> LOG.error("BlockSender.sendChunks() exception: ", e);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534943#comment-15534943
]
Takanobu Asanuma commented on HDFS-10826:
-
I created HDFS-10933 for refactoring {{TestFsck}}.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.3.patch,
> HDFS-10826.4.patch, HDFS-10826.5.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10933) Refactor TestFsck
[
https://issues.apache.org/jira/browse/HDFS-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-10933:
Description:
{{TestFsck}} should be refactored.
- use @Before @After annotations
- improve loggings
- fix checkstyle warnings
etc.
was:
{{TestFsck}} should be refactored.
- use @Before @After annotations
- fix checkstyle warnings
- improve loggings
etc.
> Refactor TestFsck
> -
>
> Key: HDFS-10933
> URL: https://issues.apache.org/jira/browse/HDFS-10933
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
>
> {{TestFsck}} should be refactored.
> - use @Before @After annotations
> - improve loggings
> - fix checkstyle warnings
> etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10933) Refactor TestFsck
Takanobu Asanuma created HDFS-10933: --- Summary: Refactor TestFsck Key: HDFS-10933 URL: https://issues.apache.org/jira/browse/HDFS-10933 Project: Hadoop HDFS Issue Type: Improvement Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma Priority: Minor TestFsck should be refactored. - use @Before @After annotations - fix checkstyle warnings - improve loggings etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10933) Refactor TestFsck
[
https://issues.apache.org/jira/browse/HDFS-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-10933:
Description:
{{TestFsck}} should be refactored.
- use @Before @After annotations
- fix checkstyle warnings
- improve loggings
etc.
was:
TestFsck should be refactored.
- use @Before @After annotations
- fix checkstyle warnings
- improve loggings
etc.
> Refactor TestFsck
> -
>
> Key: HDFS-10933
> URL: https://issues.apache.org/jira/browse/HDFS-10933
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
>
> {{TestFsck}} should be refactored.
> - use @Before @After annotations
> - fix checkstyle warnings
> - improve loggings
> etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-10826:
Attachment: HDFS-10826.5.patch
Thanks for your kind review, [~jojochuang]! I uploaded a new patch.
I agree with your thoughts, but using anotations and improving the logging will
change other methods in {{TestFsck}}. I would like to do it in a follow-on
jira. The new patch addresses the others' improvements.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.3.patch,
> HDFS-10826.4.patch, HDFS-10826.5.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10918) Add a tool to get FileEncryptionInfo from CLI
[
https://issues.apache.org/jira/browse/HDFS-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534896#comment-15534896
]
Hadoop QA commented on HDFS-10918:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
12s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
58s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.cli.TestCryptoAdminCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10918 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831025/HDFS-10918.04.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 39562a53928e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 10be459 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16937/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job
[jira] [Commented] (HDFS-10910) HDFS Erasure Coding doc should state its currently supported erasure coding policies
[
https://issues.apache.org/jira/browse/HDFS-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534887#comment-15534887
]
Hadoop QA commented on HDFS-10910:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10910 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831045/HDFS-10910.003.patch |
| Optional Tests | asflicense mvnsite |
| uname | Linux 77dd29973452 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 10be459 |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16940/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> HDFS Erasure Coding doc should state its currently supported erasure coding
> policies
>
>
> Key: HDFS-10910
> URL: https://issues.apache.org/jira/browse/HDFS-10910
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
> Attachments: HDFS-10910.001.patch, HDFS-10910.002.patch,
> HDFS-10910.003.patch
>
>
> While HDFS Erasure Coding doc states a variety of possible combinations of
> algorithms, block group size and cell size, the code (as of 3.0.0-alpha1)
> allows only three policies: RS_6_3_SCHEMA, RS_3_2_SCHEMA and
> RS_6_3_LEGACY_SCHEMA. All with default cell size. I think this should be
> documented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10907) Fix Erasure Coding documentation
[
https://issues.apache.org/jira/browse/HDFS-10907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534870#comment-15534870
]
Hadoop QA commented on HDFS-10907:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10907 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831041/HDFS-10907.01.patch |
| Optional Tests | asflicense mvnsite |
| uname | Linux 461ce4018f5d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 10be459 |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16938/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix Erasure Coding documentation
>
>
> Key: HDFS-10907
> URL: https://issues.apache.org/jira/browse/HDFS-10907
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10907.01.patch
>
>
> Found one error in HDFSErasureCoding.md, missed by HDFS-9097:
> Under
> bq. `policyName`: The ErasureCoding policy to be used for files under this
> directory. This is an optional parameter, specified using ‘-s’ flag.
> It should be '-p'.
> The same error also appears in HDFSCommands.md:
> {quote}
> Usage:
>hdfs erasurecode \[generic options]
> \[-setPolicy \[-s ] ]
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10910) HDFS Erasure Coding doc should state its currently supported erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-10910: - Attachment: HDFS-10910.003.patch Thanks [~jojochuang] for the reiview and comments. Attach a new patch to address your comments. > HDFS Erasure Coding doc should state its currently supported erasure coding > policies > > > Key: HDFS-10910 > URL: https://issues.apache.org/jira/browse/HDFS-10910 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Yiqun Lin > Attachments: HDFS-10910.001.patch, HDFS-10910.002.patch, > HDFS-10910.003.patch > > > While HDFS Erasure Coding doc states a variety of possible combinations of > algorithms, block group size and cell size, the code (as of 3.0.0-alpha1) > allows only three policies: RS_6_3_SCHEMA, RS_3_2_SCHEMA and > RS_6_3_LEGACY_SCHEMA. All with default cell size. I think this should be > documented. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10909) De-duplicate code in ErasureCodingWorker#initializeStripedReadThreadPool and DFSClient#initThreadsNumForStripedReads
[ https://issues.apache.org/jira/browse/HDFS-10909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Govindassamy updated HDFS-10909: -- Affects Version/s: 3.0.0-alpha2 Target Version/s: 3.0.0-alpha2 Status: Patch Available (was: Open) > De-duplicate code in ErasureCodingWorker#initializeStripedReadThreadPool and > DFSClient#initThreadsNumForStripedReads > > > Key: HDFS-10909 > URL: https://issues.apache.org/jira/browse/HDFS-10909 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Manoj Govindassamy >Priority: Minor > Attachments: HDFS-10909.01.patch > > > The two methods are mostly the same. Maybe it make sense to deduplicate the > code. A good place to place the code is DFSUtilClient -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10909) De-duplicate code in ErasureCodingWorker#initializeStripedReadThreadPool and DFSClient#initThreadsNumForStripedReads
[ https://issues.apache.org/jira/browse/HDFS-10909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Govindassamy updated HDFS-10909: -- Attachment: HDFS-10909.01.patch Attached patch to remove the code duplication. Made the ThreadPoolExecutor creation as a generic util which the EC readers are using now. [~jojochuang], can you please take a look ? > De-duplicate code in ErasureCodingWorker#initializeStripedReadThreadPool and > DFSClient#initThreadsNumForStripedReads > > > Key: HDFS-10909 > URL: https://issues.apache.org/jira/browse/HDFS-10909 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Manoj Govindassamy >Priority: Minor > Attachments: HDFS-10909.01.patch > > > The two methods are mostly the same. Maybe it make sense to deduplicate the > code. A good place to place the code is DFSUtilClient -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10907) Fix Erasure Coding documentation
[
https://issues.apache.org/jira/browse/HDFS-10907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-10907:
--
Status: Patch Available (was: Open)
> Fix Erasure Coding documentation
>
>
> Key: HDFS-10907
> URL: https://issues.apache.org/jira/browse/HDFS-10907
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10907.01.patch
>
>
> Found one error in HDFSErasureCoding.md, missed by HDFS-9097:
> Under
> bq. `policyName`: The ErasureCoding policy to be used for files under this
> directory. This is an optional parameter, specified using ‘-s’ flag.
> It should be '-p'.
> The same error also appears in HDFSCommands.md:
> {quote}
> Usage:
>hdfs erasurecode \[generic options]
> \[-setPolicy \[-s ] ]
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10907) Fix Erasure Coding documentation
[
https://issues.apache.org/jira/browse/HDFS-10907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-10907:
--
Attachment: HDFS-10907.01.patch
Attached patch to address EC documentation issue. [~jojochuang], can you please
take a look ?
> Fix Erasure Coding documentation
>
>
> Key: HDFS-10907
> URL: https://issues.apache.org/jira/browse/HDFS-10907
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10907.01.patch
>
>
> Found one error in HDFSErasureCoding.md, missed by HDFS-9097:
> Under
> bq. `policyName`: The ErasureCoding policy to be used for files under this
> directory. This is an optional parameter, specified using ‘-s’ flag.
> It should be '-p'.
> The same error also appears in HDFSCommands.md:
> {quote}
> Usage:
>hdfs erasurecode \[generic options]
> \[-setPolicy \[-s ] ]
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10908) Improve StripedBlockReader#createBlockReader error logging
[
https://issues.apache.org/jira/browse/HDFS-10908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-10908:
--
Affects Version/s: 3.0.0-alpha2
Target Version/s: 3.0.0-alpha2
Status: Patch Available (was: Open)
> Improve StripedBlockReader#createBlockReader error logging
> --
>
> Key: HDFS-10908
> URL: https://issues.apache.org/jira/browse/HDFS-10908
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Minor
> Attachments: HDFS-10908.01.patch
>
>
> In {{StripedBlockReader#createBlockReader}} if any IOException is thrown, the
> error is logged at debug level and then returns a null. This means in a
> typical configuration an NPE may be thrown without logging a cause if the
> StripedBlockReader object is accessed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10908) Improve StripedBlockReader#createBlockReader error logging
[
https://issues.apache.org/jira/browse/HDFS-10908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-10908:
--
Attachment: HDFS-10908.01.patch
Attaching patch to log BlockReader creation errors.
Since {{BlockReaderRemote#newBlockReader}} throws IOException with information
about the offset issues, logging the IOException should be sufficient to get
the reason for BlockReader creation errors.
{noformat}
throw new IOException("BlockReader: error in first chunk offset (" +
firstChunkOffset + ") startOffset is " +
startOffset + " for file " + file);
{noformat}
> Improve StripedBlockReader#createBlockReader error logging
> --
>
> Key: HDFS-10908
> URL: https://issues.apache.org/jira/browse/HDFS-10908
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Minor
> Attachments: HDFS-10908.01.patch
>
>
> In {{StripedBlockReader#createBlockReader}} if any IOException is thrown, the
> error is logged at debug level and then returns a null. This means in a
> typical configuration an NPE may be thrown without logging a cause if the
> StripedBlockReader object is accessed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6962) ACL inheritance conflicts with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-6962: - Attachment: test_plan.md > ACL inheritance conflicts with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch, > HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.008.patch, > HDFS-6962.009.patch, HDFS-6962.010.patch, HDFS-6962.1.patch, > disabled_new_client.log, disabled_old_client.log, enabled_new_client.log, > enabled_old_client.log, run_compat_tests, run_unit_tests, test_plan.md > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6962) ACL inheritance conflicts with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-6962: - Attachment: (was: test_plan.md) > ACL inheritance conflicts with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch, > HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.008.patch, > HDFS-6962.009.patch, HDFS-6962.010.patch, HDFS-6962.1.patch, > disabled_new_client.log, disabled_old_client.log, enabled_new_client.log, > enabled_old_client.log, run_compat_tests, run_unit_tests, test_plan.md > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10932) Ozone : fix XceiverClient slow shutdown
[
https://issues.apache.org/jira/browse/HDFS-10932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534637#comment-15534637
]
Hadoop QA commented on HDFS-10932:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HDFS-10932 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-10932 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831020/HDFS-10932.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16936/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone : fix XceiverClient slow shutdown
> ---
>
> Key: HDFS-10932
> URL: https://issues.apache.org/jira/browse/HDFS-10932
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10932.001.patch
>
>
> Currently {{XceiverClient}} is the underlying entity of
> {{DistributedStorageHandler.newKeyWriter()}} and
> {{DistributedStorageHandler.newKeyReader()}} for making call to container
> for read/write. When {{XceiverClient}} gets closed,
> {{group.shutdownGracefully()}} gets called, which is an asynchronous call.
> A problem is that this asynchronous call has default quiet period of 2
> seconds before it actually shutdown, so if we have a burst of read/write
> calls, we would end up having threads created faster than they got
> terminated, reaching system limit at some point.
> Ideally, this needs to be fixed with cached clients instead of creating new
> thread each time. This JIRA only tries to give a temporary fix for the time
> being.
> Thanks [~anu] for the offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9390) Block management for maintenance states
[
https://issues.apache.org/jira/browse/HDFS-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534504#comment-15534504
]
Hadoop QA commented on HDFS-9390:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 1067 unchanged - 15 fixed = 1067 total (was 1082) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 11s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-9390 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831013/HDFS-9390-4.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux f3510b670b7a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 10be459 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16935/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16935/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16935/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Block management for maintenance states
> ---
>
> Key: HDFS-9390
> U
[jira] [Commented] (HDFS-10931) libhdfs++: Fix object lifecycle issues in the BlockReader
[
https://issues.apache.org/jira/browse/HDFS-10931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534452#comment-15534452
]
Hadoop QA commented on HDFS-10931:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
54s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
45s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
4s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
1s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
1s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
2s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 0s{color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK
v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_111 Failed CTEST tests |
test_libhdfs_threaded_hdfspp_test_shim_static |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:78fc6b6 |
| JIRA Issue | HDFS-10931 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831009/HDFS-10931.HDFS-8707.000.patch
|
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 279f94bcfc4b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / fbba214 |
| Default Java | 1.7.0_111 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_101
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 |
| CTEST |
https://builds.apache.org/job/PreCommit-HDFS-Build/16934/artifact/patchprocess/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.7.0_111-ctest.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16934/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.7.0_111.txt
|
| JDK v1.7.0_111 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16934/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16934/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message wa
[jira] [Updated] (HDFS-10918) Add a tool to get FileEncryptionInfo from CLI
[ https://issues.apache.org/jira/browse/HDFS-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-10918: - Attachment: (was: HDFS-10918.05.patch) > Add a tool to get FileEncryptionInfo from CLI > - > > Key: HDFS-10918 > URL: https://issues.apache.org/jira/browse/HDFS-10918 > Project: Hadoop HDFS > Issue Type: New Feature > Components: encryption >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-10918.01.patch, HDFS-10918.02.patch, > HDFS-10918.03.patch, HDFS-10918.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10918) Add a tool to get FileEncryptionInfo from CLI
[ https://issues.apache.org/jira/browse/HDFS-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534438#comment-15534438 ] Xiao Chen edited comment on HDFS-10918 at 9/29/16 11:49 PM: Patch 4 to true up the docs. was (Author: xiaochen): Patch 5 to true up the docs. > Add a tool to get FileEncryptionInfo from CLI > - > > Key: HDFS-10918 > URL: https://issues.apache.org/jira/browse/HDFS-10918 > Project: Hadoop HDFS > Issue Type: New Feature > Components: encryption >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-10918.01.patch, HDFS-10918.02.patch, > HDFS-10918.03.patch, HDFS-10918.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10918) Add a tool to get FileEncryptionInfo from CLI
[ https://issues.apache.org/jira/browse/HDFS-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-10918: - Attachment: HDFS-10918.04.patch > Add a tool to get FileEncryptionInfo from CLI > - > > Key: HDFS-10918 > URL: https://issues.apache.org/jira/browse/HDFS-10918 > Project: Hadoop HDFS > Issue Type: New Feature > Components: encryption >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-10918.01.patch, HDFS-10918.02.patch, > HDFS-10918.03.patch, HDFS-10918.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10918) Add a tool to get FileEncryptionInfo from CLI
[ https://issues.apache.org/jira/browse/HDFS-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-10918: - Attachment: HDFS-10918.05.patch Patch 5 to true up the docs. > Add a tool to get FileEncryptionInfo from CLI > - > > Key: HDFS-10918 > URL: https://issues.apache.org/jira/browse/HDFS-10918 > Project: Hadoop HDFS > Issue Type: New Feature > Components: encryption >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-10918.01.patch, HDFS-10918.02.patch, > HDFS-10918.03.patch, HDFS-10918.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice
[
https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534425#comment-15534425
]
Hadoop QA commented on HDFS-10797:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 267 unchanged - 0 fixed = 270 total (was 267) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestCrcCorruption |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10797 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830965/HDFS-10797.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2e30f32b8da2 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 10be459 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16933/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16933/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16933/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16933/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Disk usage summary of snapshots causes renamed blocks to get counted twice
> --
[jira] [Commented] (HDFS-10912) Ozone:SCM: Add chill mode support to NodeManager.
[
https://issues.apache.org/jira/browse/HDFS-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534412#comment-15534412
]
Hadoop QA commented on HDFS-10912:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
6s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 5 unchanged - 1 fixed = 5 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 42s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.datanode.TestBlockPoolManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10912 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830960/HDFS-10912-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 903416ed2f3a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 595257e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16932/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16932/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16932/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone:SCM: Add chill mode support to NodeManager.
> -
>
> Key: HDFS-10912
> URL: https://issues.apache.org/jira/browse/HDFS-10912
>
[jira] [Commented] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice
[
https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534405#comment-15534405
]
Xiao Chen commented on HDFS-10797:
--
Thank you for the good discussions here [~mackrorysd] and [~jingzhao]! Sorry I
missed the scenario Jing mentioned. The semantic looks great to me.
Some nits:
- There's an used var {{count}} in {{INodeDirectory#computeContentSummary}}
- {{ContentSummaryComputationContext#nodeIncluded}} 's java doc has some typos:
{{both either}}
And more importantly, I think some update on the
{{INodeDirectory#computeContentSummary}} logic: the snapshotCounts added by
HDFS-8986 is supposed to count only contents under snapshots. Looks like this
change break the unit test from that jira. I think the {{subtreeSummary}} is
the problem here: we should only add the snapshot portion of the subtree into
snapshotCounts.
Example unit test is {{TestDFSShell#testDuSnapshots}}. What do you guys think?
> Disk usage summary of snapshots causes renamed blocks to get counted twice
> --
>
> Key: HDFS-10797
> URL: https://issues.apache.org/jira/browse/HDFS-10797
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch,
> HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch,
> HDFS-10797.006.patch
>
>
> DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how
> much disk usage is used by a snapshot by tallying up the files in the
> snapshot that have since been deleted (that way it won't overlap with regular
> files whose disk usage is computed separately). However that is determined
> from a diff that shows moved (to Trash or otherwise) or renamed files as a
> deletion and a creation operation that may overlap with the list of blocks.
> Only the deletion operation is taken into consideration, and this causes
> those blocks to get represented twice in the disk usage tallying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10932) Ozone : fix XceiverClient slow shutdown
[
https://issues.apache.org/jira/browse/HDFS-10932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534352#comment-15534352
]
Anu Engineer commented on HDFS-10932:
-
+1 , pending jenkins. I think we should have this patch until we have a
XceiverClientFactory class that can manage the caching.
> Ozone : fix XceiverClient slow shutdown
> ---
>
> Key: HDFS-10932
> URL: https://issues.apache.org/jira/browse/HDFS-10932
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10932.001.patch
>
>
> Currently {{XceiverClient}} is the underlying entity of
> {{DistributedStorageHandler.newKeyWriter()}} and
> {{DistributedStorageHandler.newKeyReader()}} for making call to container
> for read/write. When {{XceiverClient}} gets closed,
> {{group.shutdownGracefully()}} gets called, which is an asynchronous call.
> A problem is that this asynchronous call has default quiet period of 2
> seconds before it actually shutdown, so if we have a burst of read/write
> calls, we would end up having threads created faster than they got
> terminated, reaching system limit at some point.
> Ideally, this needs to be fixed with cached clients instead of creating new
> thread each time. This JIRA only tries to give a temporary fix for the time
> being.
> Thanks [~anu] for the offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10932) Ozone : fix XceiverClient slow shutdown
[
https://issues.apache.org/jira/browse/HDFS-10932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-10932:
Status: Patch Available (was: Open)
> Ozone : fix XceiverClient slow shutdown
> ---
>
> Key: HDFS-10932
> URL: https://issues.apache.org/jira/browse/HDFS-10932
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10932.001.patch
>
>
> Currently {{XceiverClient}} is the underlying entity of
> {{DistributedStorageHandler.newKeyWriter()}} and
> {{DistributedStorageHandler.newKeyReader()}} for making call to container
> for read/write. When {{XceiverClient}} gets closed,
> {{group.shutdownGracefully()}} gets called, which is an asynchronous call.
> A problem is that this asynchronous call has default quiet period of 2
> seconds before it actually shutdown, so if we have a burst of read/write
> calls, we would end up having threads created faster than they got
> terminated, reaching system limit at some point.
> Ideally, this needs to be fixed with cached clients instead of creating new
> thread each time. This JIRA only tries to give a temporary fix for the time
> being.
> Thanks [~anu] for the offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10932) Ozone : fix XceiverClient slow shutdown
[
https://issues.apache.org/jira/browse/HDFS-10932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-10932:
--
Attachment: HDFS-10932.001.patch
> Ozone : fix XceiverClient slow shutdown
> ---
>
> Key: HDFS-10932
> URL: https://issues.apache.org/jira/browse/HDFS-10932
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10932.001.patch
>
>
> Currently {{XceiverClient}} is the underlying entity of
> {{DistributedStorageHandler.newKeyWriter()}} and
> {{DistributedStorageHandler.newKeyReader()}} for making call to container
> for read/write. When {{XceiverClient}} gets closed,
> {{group.shutdownGracefully()}} gets called, which is an asynchronous call.
> A problem is that this asynchronous call has default quiet period of 2
> seconds before it actually shutdown, so if we have a burst of read/write
> calls, we would end up having threads created faster than they got
> terminated, reaching system limit at some point.
> Ideally, this needs to be fixed with cached clients instead of creating new
> thread each time. This JIRA only tries to give a temporary fix for the time
> being.
> Thanks [~anu] for the offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10906) Add unit tests for Trash with HDFS encryption zones
[ https://issues.apache.org/jira/browse/HDFS-10906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-10906: Assignee: Hanisha Koneru > Add unit tests for Trash with HDFS encryption zones > --- > > Key: HDFS-10906 > URL: https://issues.apache.org/jira/browse/HDFS-10906 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: encryption >Affects Versions: 2.8.0 >Reporter: Xiaoyu Yao >Assignee: Hanisha Koneru > > The goal is to improve unit test coverage for HDFS trash with encryption zone > especially under Kerberos environment. The current unit test > TestEncryptionZones#testEncryptionZonewithTrash() has limited coverage on > non-Kerberos case. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10932) Ozone : fix XceiverClient slow shutdown
Chen Liang created HDFS-10932:
-
Summary: Ozone : fix XceiverClient slow shutdown
Key: HDFS-10932
URL: https://issues.apache.org/jira/browse/HDFS-10932
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Chen Liang
Assignee: Chen Liang
Currently {{XceiverClient}} is the underlying entity of
{{DistributedStorageHandler.newKeyWriter()}} and
{{DistributedStorageHandler.newKeyReader()}} for making call to container for
read/write. When {{XceiverClient}} gets closed, {{group.shutdownGracefully()}}
gets called, which is an asynchronous call.
A problem is that this asynchronous call has default quiet period of 2 seconds
before it actually shutdown, so if we have a burst of read/write calls, we
would end up having threads created faster than they got terminated, reaching
system limit at some point.
Ideally, this needs to be fixed with cached clients instead of creating new
thread each time. This JIRA only tries to give a temporary fix for the time
being.
Thanks [~anu] for the offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10595) libhdfs++: Client Name Protobuf Error
[ https://issues.apache.org/jira/browse/HDFS-10595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534205#comment-15534205 ] James Clampffer commented on HDFS-10595: Looks good to me, +1. > libhdfs++: Client Name Protobuf Error > - > > Key: HDFS-10595 > URL: https://issues.apache.org/jira/browse/HDFS-10595 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Bob Hansen > Attachments: HDFS-10595.HDFS-8707.patch.000 > > > When running a cat tool > (/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cat/c/cat.c) I > get the following error: > [libprotobuf ERROR google/protobuf/wire_format.cc:1053] String field contains > invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type > if you intend to send raw bytes. > However it executes correctly. Looks like this error happens when trying to > serialize Client name in ClientOperationHeaderProto::SerializeWithCachedSizes > (/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/datatransfer.pb.cc) > Possibly the problem is caused by generating client name as a UUID in > GetRandomClientName > (/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc) > In Java client it looks like there are two different unique client > identifiers: ClientName and ClientId: > Client name is generated as: > clientName = "DFSClient_" + dfsClientConf.getTaskId() + "_" + > ThreadLocalRandom.current().nextInt() + "_" + > Thread.currentThread().getId(); > (/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java) > ClientId is generated as a UUID in > (/hadoop-common/src/main/java/org/apache/hadoop/ipc/ClientId.java) > In libhdfs++ we need to possibly also have two unique client identifiers, or > fix the current client name to work without protobuf warnings/errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9390) Block management for maintenance states
[ https://issues.apache.org/jira/browse/HDFS-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9390: -- Attachment: HDFS-9390-4.patch Updated patch to address checkstyle issues. > Block management for maintenance states > --- > > Key: HDFS-9390 > URL: https://issues.apache.org/jira/browse/HDFS-9390 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ming Ma >Assignee: Ming Ma > Attachments: HDFS-9390-2.patch, HDFS-9390-3.patch, HDFS-9390-4.patch, > HDFS-9390.patch > > > When a node is transitioned to/stay in/transitioned out of maintenance state, > we need to make sure blocks w.r.t. that nodes are properly handled. > * When nodes are put into maintenance, it will first go to > ENTERING_MAINTENANCE, and make sure blocks are minimally replicated before > the nodes are transitioned to IN_MAINTENANCE. > * Do not replica blocks when nodes are in maintenance states. Maintenance > replica will remain in BlockMaps and thus is still considered valid from > block replication point of view. In other words, putting a node to > “maintenance” mode won’t trigger BlockManager to replicate its blocks. > * Do not invalidate replicas on node under maintenance. After any file's > replication factor is reduced, NN needs to invalidate some replicas. It > should exclude nodes under maintenance in the handling. > * Do not put IN_MAINTENANCE replicas in LocatedBlock for read operation. > * Do not allocate any new block on nodes under maintenance. > * Have Balancer exclude nodes under maintenance. > * Exclude nodes under maintenance for DN cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10887) Provide admin/debug tool to dump block map
[
https://issues.apache.org/jira/browse/HDFS-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534176#comment-15534176
]
Yongjun Zhang commented on HDFS-10887:
--
Thanks a lot [~daryn].
So we can have an additional structure such as a set of DNs in NN to record the
DNs that have not reported. The set is initialized with the include list, and
when NN receives a block report from the DN, it will remove DN from the set;
when NN claims a DN to be dead (after 10.5 minutes no heartbeating), it will
add the DN to the set. Then we can have an api to report this set. This will
incur some additional cost on processing each block report (check whether the
DN exists in the set, remove if it does)
The above solution is doable when include list used, we need a way to handle
the situation when include list is not used. I guess it's rare that include
list is not used though. Do you guys agree to have the above mentioned solution
and to ignore the case that include list is not used? [~daryn] [~kihwal]?
Thanks.
> Provide admin/debug tool to dump block map
> --
>
> Key: HDFS-10887
> URL: https://issues.apache.org/jira/browse/HDFS-10887
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: hdfs, namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-10887.001.patch, HDFS-10887.002.patch
>
>
> From time to time, when NN restarts, we see
> {code}
> "The reported blocks X needs additional Y blocks to reach the threshold
> 0.9990 of total blocks Z. Safe mode will be turned off automatically.
> {code}
> We'd wonder what these blocks that still need block reports are, and what DNs
> they could possibly be located, what happened to these DNs.
> This jira to to propose a new admin or debug tool to dump the block map info
> with the blocks that have fewer than minRepl replicas.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10931) libhdfs++: Fix object lifecycle issues in the BlockReader
[ https://issues.apache.org/jira/browse/HDFS-10931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10931: --- Attachment: HDFS-10931.HDFS-8707.000.patch Patch added for the first part of the problem. Gratuitous use of shared_ptr to keep the DataNodeConnection alive. The fundamental fixes to the architecture can be addressed later on. > libhdfs++: Fix object lifecycle issues in the BlockReader > - > > Key: HDFS-10931 > URL: https://issues.apache.org/jira/browse/HDFS-10931 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-10931.HDFS-8707.000.patch > > > The BlockReader can work itself into a a state during AckRead (possibly other > stages as well) where the pipeline posts a task for asio with a pointer back > into itself, then promptly calls "delete this" without canceling the asio > request. The asio task finishes and tries to acquire the lock at the address > where the DataNodeConnection used to live - but the DN connection is no > longer valid so it's scribbling on some arbitrary bit of memory. On some > platforms the underlying address used by the mutex state will be handed out > to future mutexes so the scribble breaks that state and all the locks in that > process start misbehaving. > This can be reproduced by using the patch from HDFS-8790 and adding more > worker threads + a lot more reader threads. > I'm going to fix this in two parts: > 1) Duct tape + superglue patch to make sure that all top level continuations > in the block reader pipeline hold a shared_ptr to the DataNodeConnection. > Nested continuations also get a copy of the shared_ptr to make sure the > connection is alive. This at least keeps the connection alive so that it can > keep returning asio::operation_aborted. > 2) The continuation stuff needs a lot of work to make sure this type of bug > doesn't keep popping up. We've already fixed these issues in the RPC code. > This will most likely need to be split into a few jiras. > - Continuation "framework" can be slimmed down quite a bit, perhaps even > removed. Near zero documentation + many implied contracts = constant bug > chasing. > - Add comments to actually describe what's going on in the networking code. > This bug took significantly longer than it should have to track down because > I hadn't worked on the BlockReader in a while. > - No more "delete this". > - Flatten out nested continuations e.g. the guts of BlockReaderImpl::AckRead. > It's unclear why they were implemented like this in the first place and > there's no comments to indicate that this was intentional. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10931) libhdfs++: Fix object lifecycle issues in the BlockReader
[ https://issues.apache.org/jira/browse/HDFS-10931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10931: --- Status: Patch Available (was: Open) > libhdfs++: Fix object lifecycle issues in the BlockReader > - > > Key: HDFS-10931 > URL: https://issues.apache.org/jira/browse/HDFS-10931 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-10931.HDFS-8707.000.patch > > > The BlockReader can work itself into a a state during AckRead (possibly other > stages as well) where the pipeline posts a task for asio with a pointer back > into itself, then promptly calls "delete this" without canceling the asio > request. The asio task finishes and tries to acquire the lock at the address > where the DataNodeConnection used to live - but the DN connection is no > longer valid so it's scribbling on some arbitrary bit of memory. On some > platforms the underlying address used by the mutex state will be handed out > to future mutexes so the scribble breaks that state and all the locks in that > process start misbehaving. > This can be reproduced by using the patch from HDFS-8790 and adding more > worker threads + a lot more reader threads. > I'm going to fix this in two parts: > 1) Duct tape + superglue patch to make sure that all top level continuations > in the block reader pipeline hold a shared_ptr to the DataNodeConnection. > Nested continuations also get a copy of the shared_ptr to make sure the > connection is alive. This at least keeps the connection alive so that it can > keep returning asio::operation_aborted. > 2) The continuation stuff needs a lot of work to make sure this type of bug > doesn't keep popping up. We've already fixed these issues in the RPC code. > This will most likely need to be split into a few jiras. > - Continuation "framework" can be slimmed down quite a bit, perhaps even > removed. Near zero documentation + many implied contracts = constant bug > chasing. > - Add comments to actually describe what's going on in the networking code. > This bug took significantly longer than it should have to track down because > I hadn't worked on the BlockReader in a while. > - No more "delete this". > - Flatten out nested continuations e.g. the guts of BlockReaderImpl::AckRead. > It's unclear why they were implemented like this in the first place and > there's no comments to indicate that this was intentional. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10850) getEZForPath should NOT throw FNF
[
https://issues.apache.org/jira/browse/HDFS-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534091#comment-15534091
]
Hadoop QA commented on HDFS-10850:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
98 unchanged - 0 fixed = 99 total (was 98) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 59m
5s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10850 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830949/HDSF-10850.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e188ccd10673 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 236ac77 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16931/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16931/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16931/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://y
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534077#comment-15534077
]
Wei-Chiu Chuang commented on HDFS-10826:
Hello [~tasanuma0829] thanks for your contribution and [~ajisakaa] and
[~jingzhao] for the review. I am late to review this patch but if you don't
mind there are a little minor improvement that can be made:
in TestFsck
{code}
// make an unrecoverable ec file with corrupted blocks
for(int i = 0; i < 4; i++) {
{code}
* Instead of hardcode "4", it would be more meaningful to use parityBlocks+1 I
assume?
{code}
// Read the file to trigger reportBadBlocks
try {
IOUtils.copyBytes(fs.open(file), new
IOUtils.NullOutputStream(), conf,
true);
} catch (IOException ie) {
// Ignore exception
}
{code}
* Would it be possible to verify the exception thrown is expected when // Read
the file to trigger reportBadBlocks?
{code}
if (fs != null) {
try {
fs.close();
} catch (Exception e) {
}
}
if (cluster != null) {
cluster.shutdown();
}
{code}
You should just let the exception be thrown instead of catch and ignore it.
A better improvement could use @Before @After annotation to initialize/close
cluster and fs object. Then you do not even need to try...catch.
{code:title=waitForUnrecoverableBlockGroup()}
catch (Exception e) {
FSImage.LOG.error("Exception caught", e);
Assert.fail("Caught unexpected exception.");
}
{code}
I wonder if there is a more appropriate log object than FSImage.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
> Attachments: HDFS-10826.2.patch, HDFS-10826.3.patch,
> HDFS-10826.4.patch, HDFS-10826.WIP.1.patch
>
>
> For RS-6-3, when there is one ec block group and
> 1) 0~3 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> 2) 4~8 out of 9 internal blocks are missing, the result of fsck is HEALTY.
> {noformat}
> Erasure Coded Block Groups:
> Total size:536870912 B
> Total files: 1
> Total block groups (validated):1 (avg. block group size 536870912 B)
>
> UNRECOVERABLE BLOCK GROUPS: 1 (100.0 %)
>
> Minimally erasure-coded block groups: 0 (0.0 %)
> Over-erasure-coded block groups: 0 (0.0 %)
> Under-erasure-coded block groups: 1 (100.0 %)
> Unsatisfactory placement block groups: 0 (0.0 %)
> Default ecPolicy: RS-DEFAULT-6-3-64k
> Average block group size: 5.0
> Missing block groups: 0
> Corrupt block groups: 0
> Missing internal blocks: 4 (44.43 %)
> FSCK ended at Wed Aug 31 13:42:05 JST 2016 in 4 milliseconds
> The filesystem under path '/' is HEALTHY
> {noformat}
> 3) 9 out of 9 internal blocks are missing, the result of fsck is CRITICAL.
> (Because it is regarded as a missing block group.)
> In case 2), the result should be CRITICAL since the ec block group is
> unrecoverable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9390) Block management for maintenance states
[
https://issues.apache.org/jira/browse/HDFS-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534052#comment-15534052
]
Hadoop QA commented on HDFS-9390:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 5 new + 1067 unchanged - 15 fixed = 1072 total (was 1082) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 44s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-9390 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830933/HDFS-9390-3.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux fd4a48e3650c 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 236ac77 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16930/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16930/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16930/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16930/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
[jira] [Commented] (HDFS-10826) The result of fsck should be CRITICAL when there are unrecoverable ec block groups.
[
https://issues.apache.org/jira/browse/HDFS-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534006#comment-15534006
]
Hadoop QA commented on HDFS-10826:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 514 unchanged - 1 fixed = 515 total (was 515) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 61m
46s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10826 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830865/HDFS-10826.4.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux cb4ee846c65f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 236ac77 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16928/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16928/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16928/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> The result of fsck should be CRITICAL when there are unrecoverable ec block
> groups.
> ---
>
> Key: HDFS-10826
> URL: https://issues.apache.org/jira/browse/HDFS-10826
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>
[jira] [Commented] (HDFS-10595) libhdfs++: Client Name Protobuf Error
[
https://issues.apache.org/jira/browse/HDFS-10595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533973#comment-15533973
]
Hadoop QA commented on HDFS-10595:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
49s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
44s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
1s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
8s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
5s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
3s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
17s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with
JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_101 Failed CTEST tests |
test_libhdfs_mini_stress_hdfspp_test_shim_static |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:78fc6b6 |
| JIRA Issue | HDFS-10595 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830947/HDFS-10595.HDFS-8707.patch.000
|
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 56b607c62ace 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / fbba214 |
| Default Java | 1.7.0_111 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_101
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 |
| CTEST |
https://builds.apache.org/job/PreCommit-HDFS-Build/16929/artifact/patchprocess/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_101-ctest.txt
|
| JDK v1.7.0_111 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16929/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16
[jira] [Commented] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
[
https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533959#comment-15533959
]
Erik Krogen commented on HDFS-10921:
True, agreed that it is definitely easy to accidentally miss the requirement to
make your test path unique. Clearing the Namesystem between tests certainly
can't hurt, I would be in support of adding that.
What about something like the following to try to prevent against both
namespace pollution and the possibility of a failed test affecting subsequent
tests:
{code}
@Before
public static void resetCluster() throws Exception {
if (cluster.isClusterUp()) {
// Clear namesystem to prevent pollution issues
cluster.getNamesystem().clear();
} else {
// Previous test seems to have left cluster in a bad state;
// recreate the cluster to protect subsequent tests
cluster.shutdown();
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(REPLICATION)
.build();
cluster.waitActive();
}
}
{code}
> TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
>
>
> Key: HDFS-10921
> URL: https://issues.apache.org/jira/browse/HDFS-10921
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch
>
>
> Test fails intermittently because the NN is still in safe mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10683) Make class Token$PrivateToken private
[
https://issues.apache.org/jira/browse/HDFS-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533957#comment-15533957
]
John Zhuge commented on HDFS-10683:
---
[~jojochuang] From the test output, these 2 test failures seem unrelated to the
patch:
{noformat}
testCannotUpgradeSecondNameNode(org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA)
Time elapsed: 0.506 sec <<< ERROR!
java.net.BindException: Port in use: localhost:40448
at sun.nio.ch.Net.bind0(Native Method)
...
testStripedFileChecksumWithMissedDataBlocksRangeQuery15(org.apache.hadoop.hdfs.TestFileChecksum)
Time elapsed: 4.122 sec <<< ERROR!
java.net.BindException: Problem binding to [localhost:59525]
java.net.BindException: Address already in use; For more details see:
http://wiki.apache.org/hadoop/BindException
at sun.nio.ch.Net.bind0(Native Method)
{noformat}
> Make class Token$PrivateToken private
> -
>
> Key: HDFS-10683
> URL: https://issues.apache.org/jira/browse/HDFS-10683
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: fs, ha, security, security_token
> Attachments: HDFS-10683.001.patch, HDFS-10683.002.patch
>
>
> Avoid {{instanceof}} or typecasting of {{Toke.PrivateToken}} by introducing
> an interface method in {{Token}}. Make class {{Toke.PrivateToken}} private.
> Use a factory method instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10850) getEZForPath should NOT throw FNF
[
https://issues.apache.org/jira/browse/HDFS-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533884#comment-15533884
]
Kihwal Lee commented on HDFS-10850:
---
I tried reverting and compared with the patch. It looks like the FNFE at
{{HdfsAdmin#getEncryptionZoneForPath()}} came from HDFS-6546. The changes look
good to me.
> getEZForPath should NOT throw FNF
> -
>
> Key: HDFS-10850
> URL: https://issues.apache.org/jira/browse/HDFS-10850
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.0
>Reporter: Daryn Sharp
>Assignee: Andrew Wang
>Priority: Blocker
> Attachments: HDSF-10850.001.patch
>
>
> HDFS-9433 made an incompatible change to the semantics of getEZForPath. It
> used to return the EZ of the closest ancestor path. It never threw FNF. A
> common use of getEZForPath to determining if a file can be renamed, or must
> be copied due to mismatched EZs. Notably, this has broken hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10789) Route webhdfs through the RPC call queue
[ https://issues.apache.org/jira/browse/HDFS-10789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533868#comment-15533868 ] Kihwal Lee commented on HDFS-10789: --- The patch does not apply anymore. Please rebase/update. > Route webhdfs through the RPC call queue > > > Key: HDFS-10789 > URL: https://issues.apache.org/jira/browse/HDFS-10789 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ipc, webhdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-10789.patch > > > Webhdfs is extremely expensive under load and is not subject to the QoS > benefits of the RPC call queue. HADOOP-13537 provides the basis for routing > webhdfs through the call queue to provide unified QoS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
[
https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533839#comment-15533839
]
Rushabh S Shah edited comment on HDFS-10921 at 9/29/16 7:47 PM:
bq. the concern about namespace pollution is valid, but this is why the first
line of each of the tests is to create a Path that is unique to the given test
case and all subsequent operations occur under that Path.
I agree that the issue mentioned in this jira is not due to namespace pollution.
But this can cause test failures in future.
Lets take for example:
{{TestDiskspaceQuotaUpdate#testIncreaseReplicationBeforeCommitting}} and
{{TestDiskspaceQuotaUpdate#testDecreaseReplicationBeforeCommitting}}
Both of this test case calls {{testQuotaIssuesBeforeCommitting(short
initialReplication,short finalReplication)}} to create the file.
Here is the audit log for create call for file creation from
testIncreaseReplicationBeforeCommitting:
{noformat}
2016-09-29 11:19:02,069 [IPC Server handler 2 on 58161] INFO
FSNamesystem.audit (FSNamesystem.java:logAuditMessage(7090)) - allowed=true
ugi=rushabhs (auth:SIMPLE) ip=/127.0.0.1 cmd=create
src=/TestQuotaUpdate/testQuotaIssuesBeforeCommitting/1-4/testfile
dst=nullperm=rushabhs:supergroup:rw-r--r-- proto=rpc
{noformat}
Here is the audit log for create call for file creation from
testDecreaseReplicationBeforeCommitting:
{noformat}
2016-09-29 11:20:20,403 [IPC Server handler 3 on 58161] INFO
FSNamesystem.audit (FSNamesystem.java:logAuditMessage(7090)) - allowed=true
ugi=rushabhs (auth:SIMPLE) ip=/127.0.0.1 cmd=create
src=/TestQuotaUpdate/testQuotaIssuesBeforeCommitting/4-1/testfile
dst=nullperm=rushabhs:supergroup:rw-r--r-- proto=rpc
{noformat}
Only difference between 2 file paths is the
{{initialReplication-finalReplication}} value.
And we can't expect every developers in future to do the right thing while
creating the file.
I can point out at least 100's of file creations in other test suites that
don't create a path that is unique to test case.
That's why I think we should clear the namesystem state before running new test
case.
was (Author: shahrs87):
bq. the concern about namespace pollution is valid, but this is why the first
line of each of the tests is to create a Path that is unique to the given test
case and all subsequent operations occur under that Path.
I agree that the issue mentioned in this jira is not due to namespace pollution.
But this can cause test failures in future.
Lets take for example:
{{TestDiskspaceQuotaUpdate#testIncreaseReplicationBeforeCommitting}} and
{{TestDiskspaceQuotaUpdate#testDecreaseReplicationBeforeCommitting}}
Both of this test case calls {{testQuotaIssuesBeforeCommitting(short
initialReplication,short finalReplication)}} to create the file.
Here is the audit log for create call for file creation from
testIncreaseReplicationBeforeCommitting:
{noformat}
2016-09-29 11:19:02,069 [IPC Server handler 2 on 58161] INFO
FSNamesystem.audit (FSNamesystem.java:logAuditMessage(7090)) - allowed=true
ugi=rushabhs (auth:SIMPLE) ip=/127.0.0.1 cmd=create
src=/TestQuotaUpdate/testQuotaIssuesBeforeCommitting/1-4/testfile
dst=nullperm=rushabhs:supergroup:rw-r--r-- proto=rpc
{noformat}
Here is the audit log for create call for file creation from
testDecreaseReplicationBeforeCommitting:
{noformat}
2016-09-29 11:20:20,403 [IPC Server handler 3 on 58161] INFO
FSNamesystem.audit (FSNamesystem.java:logAuditMessage(7090)) - allowed=true
ugi=rushabhs (auth:SIMPLE) ip=/127.0.0.1 cmd=create
src=/TestQuotaUpdate/testQuotaIssuesBeforeCommitting/4-1/testfile
dst=nullperm=rushabhs:supergroup:rw-r--r-- proto=rpc
{noformat}
Only difference between 2 file creations is the
{{initialReplication-finalReplication}} value.
And we can't expect every developers in future to do the right thing while
creating the file.
I can point out at least 100's of file creations in other test suites that
don't create a path that is unique to test case.
That's why I think we should clear the namesystem state before running new test
case.
> TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
>
>
> Key: HDFS-10921
> URL: https://issues.apache.org/jira/browse/HDFS-10921
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch
>
>
> Test fails intermittently because the NN is still in safe mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail:
[jira] [Commented] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
[
https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533839#comment-15533839
]
Rushabh S Shah commented on HDFS-10921:
---
bq. the concern about namespace pollution is valid, but this is why the first
line of each of the tests is to create a Path that is unique to the given test
case and all subsequent operations occur under that Path.
I agree that the issue mentioned in this jira is not due to namespace pollution.
But this can cause test failures in future.
Lets take for example:
{{TestDiskspaceQuotaUpdate#testIncreaseReplicationBeforeCommitting}} and
{{TestDiskspaceQuotaUpdate#testDecreaseReplicationBeforeCommitting}}
Both of this test case calls {{testQuotaIssuesBeforeCommitting(short
initialReplication,short finalReplication)}} to create the file.
Here is the audit log for create call for file creation from
testIncreaseReplicationBeforeCommitting:
{noformat}
2016-09-29 11:19:02,069 [IPC Server handler 2 on 58161] INFO
FSNamesystem.audit (FSNamesystem.java:logAuditMessage(7090)) - allowed=true
ugi=rushabhs (auth:SIMPLE) ip=/127.0.0.1 cmd=create
src=/TestQuotaUpdate/testQuotaIssuesBeforeCommitting/1-4/testfile
dst=nullperm=rushabhs:supergroup:rw-r--r-- proto=rpc
{noformat}
Here is the audit log for create call for file creation from
testDecreaseReplicationBeforeCommitting:
{noformat}
2016-09-29 11:20:20,403 [IPC Server handler 3 on 58161] INFO
FSNamesystem.audit (FSNamesystem.java:logAuditMessage(7090)) - allowed=true
ugi=rushabhs (auth:SIMPLE) ip=/127.0.0.1 cmd=create
src=/TestQuotaUpdate/testQuotaIssuesBeforeCommitting/4-1/testfile
dst=nullperm=rushabhs:supergroup:rw-r--r-- proto=rpc
{noformat}
Only difference between 2 file creations is the
{{initialReplication-finalReplication}} value.
And we can't expect every developers in future to do the right thing while
creating the file.
I can point out at least 100's of file creations in other test suites that
don't create a path that is unique to test case.
That's why I think we should clear the namesystem state before running new test
case.
> TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
>
>
> Key: HDFS-10921
> URL: https://issues.apache.org/jira/browse/HDFS-10921
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch
>
>
> Test fails intermittently because the NN is still in safe mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
[
https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533813#comment-15533813
]
Erik Krogen commented on HDFS-10921:
This is certainly true and something I discussed with Konstantin before making
the change. We decided to go for decreased runtime. I wonder if a pattern could
be developed to help with this. e.g., in a {{@Before}} block, check cluster
health ({{isClusterUp()}}) and only if that fails then create a new cluster for
subsequent tests?
> TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
>
>
> Key: HDFS-10921
> URL: https://issues.apache.org/jira/browse/HDFS-10921
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch
>
>
> Test fails intermittently because the NN is still in safe mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10931) libhdfs++: Fix object lifecycle issues in the BlockReader
James Clampffer created HDFS-10931: -- Summary: libhdfs++: Fix object lifecycle issues in the BlockReader Key: HDFS-10931 URL: https://issues.apache.org/jira/browse/HDFS-10931 Project: Hadoop HDFS Issue Type: Sub-task Reporter: James Clampffer Assignee: James Clampffer Priority: Critical The BlockReader can work itself into a a state during AckRead (possibly other stages as well) where the pipeline posts a task for asio with a pointer back into itself, then promptly calls "delete this" without canceling the asio request. The asio task finishes and tries to acquire the lock at the address where the DataNodeConnection used to live - but the DN connection is no longer valid so it's scribbling on some arbitrary bit of memory. On some platforms the underlying address used by the mutex state will be handed out to future mutexes so the scribble breaks that state and all the locks in that process start misbehaving. This can be reproduced by using the patch from HDFS-8790 and adding more worker threads + a lot more reader threads. I'm going to fix this in two parts: 1) Duct tape + superglue patch to make sure that all top level continuations in the block reader pipeline hold a shared_ptr to the DataNodeConnection. Nested continuations also get a copy of the shared_ptr to make sure the connection is alive. This at least keeps the connection alive so that it can keep returning asio::operation_aborted. 2) The continuation stuff needs a lot of work to make sure this type of bug doesn't keep popping up. We've already fixed these issues in the RPC code. This will most likely need to be split into a few jiras. - Continuation "framework" can be slimmed down quite a bit, perhaps even removed. Near zero documentation + many implied contracts = constant bug chasing. - Add comments to actually describe what's going on in the networking code. This bug took significantly longer than it should have to track down because I hadn't worked on the BlockReader in a while. - No more "delete this". - Flatten out nested continuations e.g. the guts of BlockReaderImpl::AckRead. It's unclear why they were implemented like this in the first place and there's no comments to indicate that this was intentional. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice
[ https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HDFS-10797: - Attachment: HDFS-10797.006.patch Great! During manual testing, I found a bug where if you snapshot a file, append to it and then rename it, space consumed suddenly decreases: the reference to the "deleted" node in the snapshot diff was getting counted first, and the current state of the file was then ignored. I changed the approach so summarization of any deleted nodes is deferred until the end - then we can choose whether or not to include them only once we have all the context we need. Also added a test case that would've caught the bug I found and cleaned up the code a bunch. > Disk usage summary of snapshots causes renamed blocks to get counted twice > -- > > Key: HDFS-10797 > URL: https://issues.apache.org/jira/browse/HDFS-10797 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch, > HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch, > HDFS-10797.006.patch > > > DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how > much disk usage is used by a snapshot by tallying up the files in the > snapshot that have since been deleted (that way it won't overlap with regular > files whose disk usage is computed separately). However that is determined > from a diff that shows moved (to Trash or otherwise) or renamed files as a > deletion and a creation operation that may overlap with the list of blocks. > Only the deletion operation is taken into consideration, and this causes > those blocks to get represented twice in the disk usage tallying. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice
[ https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HDFS-10797: - Status: Patch Available (was: Open) > Disk usage summary of snapshots causes renamed blocks to get counted twice > -- > > Key: HDFS-10797 > URL: https://issues.apache.org/jira/browse/HDFS-10797 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch, > HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch, > HDFS-10797.006.patch > > > DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how > much disk usage is used by a snapshot by tallying up the files in the > snapshot that have since been deleted (that way it won't overlap with regular > files whose disk usage is computed separately). However that is determined > from a diff that shows moved (to Trash or otherwise) or renamed files as a > deletion and a creation operation that may overlap with the list of blocks. > Only the deletion operation is taken into consideration, and this causes > those blocks to get represented twice in the disk usage tallying. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10429) DataStreamer interrupted warning always appears when using CLI upload file
[
https://issues.apache.org/jira/browse/HDFS-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533730#comment-15533730
]
Hadoop QA commented on HDFS-10429:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10429 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804790/HDFS-10429.1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 06b14025bdd5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 236ac77 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16927/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16927/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> DataStreamer interrupted warning always appears when using CLI upload file
> ---
>
> Key: HDFS-10429
> URL: https://issues.apache.org/jira/browse/HDFS-10429
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
>Priority: Minor
> Attachments: HDFS-104
[jira] [Updated] (HDFS-10912) Ozone:SCM: Add chill mode support to NodeManager.
[ https://issues.apache.org/jira/browse/HDFS-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-10912: Status: Patch Available (was: Open) > Ozone:SCM: Add chill mode support to NodeManager. > - > > Key: HDFS-10912 > URL: https://issues.apache.org/jira/browse/HDFS-10912 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-10912-HDFS-7240.001.patch, > HDFS-10912-HDFS-7240.002.patch > > > Add Safe mode support : That is add the ability to force exit or enter safe > mode. As well as get the current safe mode status from node manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10912) Ozone:SCM: Add chill mode support to NodeManager.
[ https://issues.apache.org/jira/browse/HDFS-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-10912: Attachment: HDFS-10912-HDFS-7240.002.patch Renaming safe mode to chill mode to indicate this is the period before node manager gets into action. To see historical references to chill mode, please google "LeBron James chill mode". > Ozone:SCM: Add chill mode support to NodeManager. > - > > Key: HDFS-10912 > URL: https://issues.apache.org/jira/browse/HDFS-10912 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-10912-HDFS-7240.001.patch, > HDFS-10912-HDFS-7240.002.patch > > > Add Safe mode support : That is add the ability to force exit or enter safe > mode. As well as get the current safe mode status from node manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice
[ https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533611#comment-15533611 ] Jing Zhao commented on HDFS-10797: -- Actually my proposal is like your .005 patch. The current semantic and approach looks good to me overall. > Disk usage summary of snapshots causes renamed blocks to get counted twice > -- > > Key: HDFS-10797 > URL: https://issues.apache.org/jira/browse/HDFS-10797 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch, > HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch > > > DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how > much disk usage is used by a snapshot by tallying up the files in the > snapshot that have since been deleted (that way it won't overlap with regular > files whose disk usage is computed separately). However that is determined > from a diff that shows moved (to Trash or otherwise) or renamed files as a > deletion and a creation operation that may overlap with the list of blocks. > Only the deletion operation is taken into consideration, and this causes > those blocks to get represented twice in the disk usage tallying. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10912) Ozone:SCM: Add chill mode support to NodeManager.
[ https://issues.apache.org/jira/browse/HDFS-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-10912: Summary: Ozone:SCM: Add chill mode support to NodeManager. (was: Ozone:SCM: Add safe mode support to NodeManager.) > Ozone:SCM: Add chill mode support to NodeManager. > - > > Key: HDFS-10912 > URL: https://issues.apache.org/jira/browse/HDFS-10912 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-10912-HDFS-7240.001.patch > > > Add Safe mode support : That is add the ability to force exit or enter safe > mode. As well as get the current safe mode status from node manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10897) Ozone: SCM: Add NodeManager
[ https://issues.apache.org/jira/browse/HDFS-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-10897: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I have committed this to the feature branch. > Ozone: SCM: Add NodeManager > --- > > Key: HDFS-10897 > URL: https://issues.apache.org/jira/browse/HDFS-10897 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-10897-HDFS-7240.001.patch, > HDFS-10897-HDFS-7240.002.patch, HDFS-10897-HDFS-7240.003.patch > > > Add a nodeManager class that will be used by Storage Controller Manager > eventually. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10916) Switch from "raw" to "system" xattr namespace for erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-10916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533586#comment-15533586 ] Andrew Wang commented on HDFS-10916: [~zhz] you mind doing the quick review? > Switch from "raw" to "system" xattr namespace for erasure coding policy > --- > > Key: HDFS-10916 > URL: https://issues.apache.org/jira/browse/HDFS-10916 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-10916.001.patch > > > Currently EC policy is stored as in the raw xattr namespace. It would be > better to store this in "system" like storage policy. > Raw is meant for attributes which need to be preserved across a distcp, like > encryption info. EC policy is more similar to replication factor or storage > policy, which can differ between the src and target of a distcp. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10609) Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications
[
https://issues.apache.org/jira/browse/HDFS-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533581#comment-15533581
]
Hadoop QA commented on HDFS-10609:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
31s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
12s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
58s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 584 unchanged - 5 fixed = 587 total (was 589) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2943 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 1m
13s{color} | {color:red} The patch 113 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 5s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_111. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
19s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_101 Failed junit tests | hadoop.hdfs.web.TestWebHDFS |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| JDK v1.8.0_101 Timed out junit tests |
org.apache.hadoop.hdfs.web.TestWebHdfsTokens |
| JDK v1.7.0_111 Failed junit tests | hadoop.hdfs.web.TestWebHdfsTokens |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:c420dfe
[jira] [Commented] (HDFS-10918) Add a tool to get FileEncryptionInfo from CLI
[ https://issues.apache.org/jira/browse/HDFS-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533578#comment-15533578 ] Andrew Wang commented on HDFS-10918: Great, this looks right. Some last little nits: * In the doc, we should also add a table with the arguments like the other commands. * With just this help text, it's not clear what info is returned by this command and why a user might care. Consider expanding the help text about usecases, and adding sample output to the doc. +1 pending though, looks good overall. > Add a tool to get FileEncryptionInfo from CLI > - > > Key: HDFS-10918 > URL: https://issues.apache.org/jira/browse/HDFS-10918 > Project: Hadoop HDFS > Issue Type: New Feature > Components: encryption >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-10918.01.patch, HDFS-10918.02.patch, > HDFS-10918.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10850) getEZForPath should NOT throw FNF
[ https://issues.apache.org/jira/browse/HDFS-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10850: --- Assignee: Andrew Wang (was: Rakesh R) Target Version/s: 2.8.0, 3.0.0-alpha2 (was: 2.8.0) Status: Patch Available (was: Open) > getEZForPath should NOT throw FNF > - > > Key: HDFS-10850 > URL: https://issues.apache.org/jira/browse/HDFS-10850 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Andrew Wang >Priority: Blocker > Attachments: HDSF-10850.001.patch > > > HDFS-9433 made an incompatible change to the semantics of getEZForPath. It > used to return the EZ of the closest ancestor path. It never threw FNF. A > common use of getEZForPath to determining if a file can be renamed, or must > be copied due to mismatched EZs. Notably, this has broken hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10850) getEZForPath should NOT throw FNF
[ https://issues.apache.org/jira/browse/HDFS-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10850: --- Attachment: HDSF-10850.001.patch Rakesh, hope you don't mind, but I spent a little time making a new patch based on the revert. Besides reverting, I also updated the javadoc for HdfsAdmin and modified the test to assert the "returns null for non-existent path" behavior. > getEZForPath should NOT throw FNF > - > > Key: HDFS-10850 > URL: https://issues.apache.org/jira/browse/HDFS-10850 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Rakesh R >Priority: Blocker > Attachments: HDSF-10850.001.patch > > > HDFS-9433 made an incompatible change to the semantics of getEZForPath. It > used to return the EZ of the closest ancestor path. It never threw FNF. A > common use of getEZForPath to determining if a file can be renamed, or must > be copied due to mismatched EZs. Notably, this has broken hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10595) libhdfs++: Client Name Protobuf Error
[ https://issues.apache.org/jira/browse/HDFS-10595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen updated HDFS-10595: -- Status: Patch Available (was: Reopened) > libhdfs++: Client Name Protobuf Error > - > > Key: HDFS-10595 > URL: https://issues.apache.org/jira/browse/HDFS-10595 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Bob Hansen > Attachments: HDFS-10595.HDFS-8707.patch.000 > > > When running a cat tool > (/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cat/c/cat.c) I > get the following error: > [libprotobuf ERROR google/protobuf/wire_format.cc:1053] String field contains > invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type > if you intend to send raw bytes. > However it executes correctly. Looks like this error happens when trying to > serialize Client name in ClientOperationHeaderProto::SerializeWithCachedSizes > (/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/datatransfer.pb.cc) > Possibly the problem is caused by generating client name as a UUID in > GetRandomClientName > (/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc) > In Java client it looks like there are two different unique client > identifiers: ClientName and ClientId: > Client name is generated as: > clientName = "DFSClient_" + dfsClientConf.getTaskId() + "_" + > ThreadLocalRandom.current().nextInt() + "_" + > Thread.currentThread().getId(); > (/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java) > ClientId is generated as a UUID in > (/hadoop-common/src/main/java/org/apache/hadoop/ipc/ClientId.java) > In libhdfs++ we need to possibly also have two unique client identifiers, or > fix the current client name to work without protobuf warnings/errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10595) libhdfs++: Client Name Protobuf Error
[ https://issues.apache.org/jira/browse/HDFS-10595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen updated HDFS-10595: -- Attachment: HDFS-10595.HDFS-8707.patch.000 > libhdfs++: Client Name Protobuf Error > - > > Key: HDFS-10595 > URL: https://issues.apache.org/jira/browse/HDFS-10595 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Bob Hansen > Attachments: HDFS-10595.HDFS-8707.patch.000 > > > When running a cat tool > (/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cat/c/cat.c) I > get the following error: > [libprotobuf ERROR google/protobuf/wire_format.cc:1053] String field contains > invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type > if you intend to send raw bytes. > However it executes correctly. Looks like this error happens when trying to > serialize Client name in ClientOperationHeaderProto::SerializeWithCachedSizes > (/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/datatransfer.pb.cc) > Possibly the problem is caused by generating client name as a UUID in > GetRandomClientName > (/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc) > In Java client it looks like there are two different unique client > identifiers: ClientName and ClientId: > Client name is generated as: > clientName = "DFSClient_" + dfsClientConf.getTaskId() + "_" + > ThreadLocalRandom.current().nextInt() + "_" + > Thread.currentThread().getId(); > (/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java) > ClientId is generated as a UUID in > (/hadoop-common/src/main/java/org/apache/hadoop/ipc/ClientId.java) > In libhdfs++ we need to possibly also have two unique client identifiers, or > fix the current client name to work without protobuf warnings/errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
[ https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533525#comment-15533525 ] Eric Badger commented on HDFS-10921: bq. Can we just change the calls to cluster.restartNameNodes() to cluster.restartNameNode(true)? [~xkrogen], that sounds like a reasonable change. The only reservation I have with that is that I've encountered problems when tests use @BeforeClass to start up a cluster and use it for all tests. If one of the tests fails and kills the cluster, then all subsequent tests will fail because they depended on the cluster being left in a usable state. I've seen 20 tests fail in a class because a single test failed. Maybe that's an acceptable tradeoff for decreased runtime, but it makes debugging the specific failure harder. > TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode > > > Key: HDFS-10921 > URL: https://issues.apache.org/jira/browse/HDFS-10921 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch > > > Test fails intermittently because the NN is still in safe mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-10595) libhdfs++: Client Name Protobuf Error
[ https://issues.apache.org/jira/browse/HDFS-10595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen reopened HDFS-10595: --- Assignee: Bob Hansen On second thought, let me fix this issue here and see if the rest of the errors go away in HDFS-9453. > libhdfs++: Client Name Protobuf Error > - > > Key: HDFS-10595 > URL: https://issues.apache.org/jira/browse/HDFS-10595 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Bob Hansen > > When running a cat tool > (/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cat/c/cat.c) I > get the following error: > [libprotobuf ERROR google/protobuf/wire_format.cc:1053] String field contains > invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type > if you intend to send raw bytes. > However it executes correctly. Looks like this error happens when trying to > serialize Client name in ClientOperationHeaderProto::SerializeWithCachedSizes > (/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/datatransfer.pb.cc) > Possibly the problem is caused by generating client name as a UUID in > GetRandomClientName > (/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc) > In Java client it looks like there are two different unique client > identifiers: ClientName and ClientId: > Client name is generated as: > clientName = "DFSClient_" + dfsClientConf.getTaskId() + "_" + > ThreadLocalRandom.current().nextInt() + "_" + > Thread.currentThread().getId(); > (/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java) > ClientId is generated as a UUID in > (/hadoop-common/src/main/java/org/apache/hadoop/ipc/ClientId.java) > In libhdfs++ we need to possibly also have two unique client identifiers, or > fix the current client name to work without protobuf warnings/errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-10595) libhdfs++: Client Name Protobuf Error
[ https://issues.apache.org/jira/browse/HDFS-10595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen resolved HDFS-10595. --- Resolution: Duplicate Dupe of HDFS-9453 > libhdfs++: Client Name Protobuf Error > - > > Key: HDFS-10595 > URL: https://issues.apache.org/jira/browse/HDFS-10595 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein > > When running a cat tool > (/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cat/c/cat.c) I > get the following error: > [libprotobuf ERROR google/protobuf/wire_format.cc:1053] String field contains > invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type > if you intend to send raw bytes. > However it executes correctly. Looks like this error happens when trying to > serialize Client name in ClientOperationHeaderProto::SerializeWithCachedSizes > (/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/datatransfer.pb.cc) > Possibly the problem is caused by generating client name as a UUID in > GetRandomClientName > (/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc) > In Java client it looks like there are two different unique client > identifiers: ClientName and ClientId: > Client name is generated as: > clientName = "DFSClient_" + dfsClientConf.getTaskId() + "_" + > ThreadLocalRandom.current().nextInt() + "_" + > Thread.currentThread().getId(); > (/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java) > ClientId is generated as a UUID in > (/hadoop-common/src/main/java/org/apache/hadoop/ipc/ClientId.java) > In libhdfs++ we need to possibly also have two unique client identifiers, or > fix the current client name to work without protobuf warnings/errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10910) HDFS Erasure Coding doc should state its currently supported erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533474#comment-15533474 ] Wei-Chiu Chuang commented on HDFS-10910: Actually, to be accurate, we should not just spell out the schema name. Instead, the policy names are RS-DEFAULT-3-2-64k, RS-DEFAULT-6-3-64k and RS-LEGACY-6-3-64k. This is how you specify ec policy using hdfs erasurecode -setPolicy -p > HDFS Erasure Coding doc should state its currently supported erasure coding > policies > > > Key: HDFS-10910 > URL: https://issues.apache.org/jira/browse/HDFS-10910 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Yiqun Lin > Attachments: HDFS-10910.001.patch, HDFS-10910.002.patch > > > While HDFS Erasure Coding doc states a variety of possible combinations of > algorithms, block group size and cell size, the code (as of 3.0.0-alpha1) > allows only three policies: RS_6_3_SCHEMA, RS_3_2_SCHEMA and > RS_6_3_LEGACY_SCHEMA. All with default cell size. I think this should be > documented. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6708) StorageType should be encoded in the block token
[ https://issues.apache.org/jira/browse/HDFS-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533473#comment-15533473 ] Arpit Agarwal commented on HDFS-6708: - I added you as a project contributor and assigned it to you. Proper typing will be an incompatible change and probably too disruptive to be practical. IIUC there's many dependencies across HDFS/Yarn that would need to be updated. > StorageType should be encoded in the block token > > > Key: HDFS-6708 > URL: https://issues.apache.org/jira/browse/HDFS-6708 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.4.1 >Reporter: Arpit Agarwal >Assignee: Pieter Reuse > > HDFS-6702 is adding support for file creation based on StorageType. > The block token is used as a tamper-proof channel for communicating block > parameters from the NN to the DN during block creation. The StorageType > should be included in this block token. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10850) getEZForPath should NOT throw FNF
[ https://issues.apache.org/jira/browse/HDFS-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533459#comment-15533459 ] Andrew Wang commented on HDFS-10850: Okay, sounds good. Since this was released in 3.0.0-alpha1, let's use this JIRA to track the revert for changelog purposes. We can open a new JIRA for the special rename exception. > getEZForPath should NOT throw FNF > - > > Key: HDFS-10850 > URL: https://issues.apache.org/jira/browse/HDFS-10850 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Rakesh R >Priority: Blocker > > HDFS-9433 made an incompatible change to the semantics of getEZForPath. It > used to return the EZ of the closest ancestor path. It never threw FNF. A > common use of getEZForPath to determining if a file can be renamed, or must > be copied due to mismatched EZs. Notably, this has broken hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6708) StorageType should be encoded in the block token
[ https://issues.apache.org/jira/browse/HDFS-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6708: Assignee: Pieter Reuse > StorageType should be encoded in the block token > > > Key: HDFS-6708 > URL: https://issues.apache.org/jira/browse/HDFS-6708 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.4.1 >Reporter: Arpit Agarwal >Assignee: Pieter Reuse > > HDFS-6702 is adding support for file creation based on StorageType. > The block token is used as a tamper-proof channel for communicating block > parameters from the NN to the DN during block creation. The StorageType > should be included in this block token. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9390) Block management for maintenance states
[
https://issues.apache.org/jira/browse/HDFS-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-9390:
--
Attachment: HDFS-9390-3.patch
Thanks [~eddyxu] for the review! Here is the new patch.
bq. If we change it to the following code, we can undo most of the
DatanodeManager.java changes, of which the motivation of these changes are not
clear to me in the first sight.
The main reason is {{DatanodeManager#removeDatanode}} performs other operations
such as {{heartbeatManager.removeDatanode(nodeInfo);}} and
{{blockManager.getBlockReportLeaseManager().unregister(nodeInfo);}} which
should be called when a maintenance node becomes dead.
bq. Why it does not re-calculate stats when minReplicationToBeInMaintanence ==
0?
Good catch. In addition to fixing it, the new patch also updates
TestNamenodeCapacityReport to cover maintenance scenario.
bq. Is the comment correct in the context?
Fixed.
bq. One related question is that, why startMaintenance() and stopMaintenance()
are in DecommissionManager.
This is similar to startDecommission() and stopDecommission() in
DecommissionManager. I plan to rename DecommissionManager to
AdminServiceManager as part of HDFS-9388.
bq. In NumberReplicas.java, you might want consider rename int maintenance() to
int maintenanceReplicas, so is liveEnteringMaintence().
Fixed.
> Block management for maintenance states
> ---
>
> Key: HDFS-9390
> URL: https://issues.apache.org/jira/browse/HDFS-9390
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-9390-2.patch, HDFS-9390-3.patch, HDFS-9390.patch
>
>
> When a node is transitioned to/stay in/transitioned out of maintenance state,
> we need to make sure blocks w.r.t. that nodes are properly handled.
> * When nodes are put into maintenance, it will first go to
> ENTERING_MAINTENANCE, and make sure blocks are minimally replicated before
> the nodes are transitioned to IN_MAINTENANCE.
> * Do not replica blocks when nodes are in maintenance states. Maintenance
> replica will remain in BlockMaps and thus is still considered valid from
> block replication point of view. In other words, putting a node to
> “maintenance” mode won’t trigger BlockManager to replicate its blocks.
> * Do not invalidate replicas on node under maintenance. After any file's
> replication factor is reduced, NN needs to invalidate some replicas. It
> should exclude nodes under maintenance in the handling.
> * Do not put IN_MAINTENANCE replicas in LocatedBlock for read operation.
> * Do not allocate any new block on nodes under maintenance.
> * Have Balancer exclude nodes under maintenance.
> * Exclude nodes under maintenance for DN cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10912) Ozone:SCM: Add safe mode support to NodeManager.
[ https://issues.apache.org/jira/browse/HDFS-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533407#comment-15533407 ] Anu Engineer commented on HDFS-10912: - My options at this moment is to name this as "nonSafeSafemode" or chillMode. I think chillMode looks better. Anyways we can always rename these modes -- Hopefully all these names are hidden inside NodeManager so that no one really cares. So In short, I will rename this to chill mode for now and wait until someone tells me how badly it is named and comes up with a better name :) Also when we have real safe mode, that does take look at the container state, this function would be used from that mode. So this chill mode is not going to be public. > Ozone:SCM: Add safe mode support to NodeManager. > > > Key: HDFS-10912 > URL: https://issues.apache.org/jira/browse/HDFS-10912 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-10912-HDFS-7240.001.patch > > > Add Safe mode support : That is add the ability to force exit or enter safe > mode. As well as get the current safe mode status from node manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
[
https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533396#comment-15533396
]
Erik Krogen commented on HDFS-10921:
[~shahrs87], the concern about namespace pollution is valid, but this is why
the first line of each of the tests is to create a Path that is unique to the
given test case and all subsequent operations occur under that Path.
Especially given the error that Eric has posted, I don't think that is the
issue.
The reason that {{cluster.restartNameNodes()}} is called, IIUC, is to initiate
a check of the edit log to ensure that edits aren't corrupted as per the
comment right above the call - previously the cluster was completely recreated
after each test case so there would be no reason (in terms of cleaning) to
restart the namenode at the end of the test. Pinging [~jingzhao] to confirm.
[~ebadger], first off, good catch. Apologies for introducing this bug. Can we
just change the calls to {{cluster.restartNameNodes()}} to
{{cluster.restartNameNode(true)}}? There is only one NN in this test so we can
just use the already-available method on {{MiniDFSCluster}} and keep the change
local to the test itself.
> TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
>
>
> Key: HDFS-10921
> URL: https://issues.apache.org/jira/browse/HDFS-10921
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch
>
>
> Test fails intermittently because the NN is still in safe mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
[
https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533396#comment-15533396
]
Erik Krogen edited comment on HDFS-10921 at 9/29/16 5:22 PM:
-
[~shahrs87], the concern about namespace pollution is valid, but this is why
the first line of each of the tests is to create a Path that is unique to the
given test case and all subsequent operations occur under that Path.
Especially given the error that Eric has posted, I don't think that is the
issue.
The reason that {{cluster.restartNameNodes()}} is called, IIUC, is to initiate
a check of the edit log to ensure that edits aren't corrupted as per the
comment right above the call - previously the cluster was completely recreated
after each test case so there would be no reason (in terms of cleaning) to
restart the namenode at the end of the test. Pinging [~jingzhao] to confirm.
[~ebadger], first off, good catch. Apologies for introducing this issue and
thank you for helping to deal with it. Can we just change the calls to
{{cluster.restartNameNodes()}} to {{cluster.restartNameNode(true)}}? There is
only one NN in this test so we can just use the already-available method on
{{MiniDFSCluster}} and keep the change local to the test itself.
was (Author: xkrogen):
[~shahrs87], the concern about namespace pollution is valid, but this is why
the first line of each of the tests is to create a Path that is unique to the
given test case and all subsequent operations occur under that Path.
Especially given the error that Eric has posted, I don't think that is the
issue.
The reason that {{cluster.restartNameNodes()}} is called, IIUC, is to initiate
a check of the edit log to ensure that edits aren't corrupted as per the
comment right above the call - previously the cluster was completely recreated
after each test case so there would be no reason (in terms of cleaning) to
restart the namenode at the end of the test. Pinging [~jingzhao] to confirm.
[~ebadger], first off, good catch. Apologies for introducing this bug. Can we
just change the calls to {{cluster.restartNameNodes()}} to
{{cluster.restartNameNode(true)}}? There is only one NN in this test so we can
just use the already-available method on {{MiniDFSCluster}} and keep the change
local to the test itself.
> TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
>
>
> Key: HDFS-10921
> URL: https://issues.apache.org/jira/browse/HDFS-10921
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch
>
>
> Test fails intermittently because the NN is still in safe mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10930) Refactor Datanode IO operations into a single class
Xiaoyu Yao created HDFS-10930: - Summary: Refactor Datanode IO operations into a single class Key: HDFS-10930 URL: https://issues.apache.org/jira/browse/HDFS-10930 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Datanode IO (Disk/Network) related operations and instrumentations are currently spilled in many classes such as DataNode.java, BlockReceiver.java, BlockSender.java, FsDatasetImpl.java, FsVolumeImpl.java, DirectoryScanner.java, BlockScanner.java, FsDatasetAsyncDiskService.java, LocalReplica.java, LocalReplicaPipeline.java, Storage.java, etc. This ticket is opened to consolidate them into a single class. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10910) HDFS Erasure Coding doc should state its currently supported erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533383#comment-15533383 ] Wei-Chiu Chuang edited comment on HDFS-10910 at 9/29/16 5:18 PM: - Thanks a lot [~linyiqun] for submitting the patch and thanks [~Sammi] for the first review. I think the patch is mostly good, just one nit for clarity: bq. The current codec algorithms support three policies: To be more specific, you may say "There are three policies currently being supported". Because a policy constitutes a schema (=coding algorithm + data unit and parity unit) and a cell size. was (Author: jojochuang): Thanks a lot [~linyiqun] for submitting the patch and thanks [~Sammi] for the first review. I think the patch is mostly good, just one nit for clarity: bq. The current codec algorithms support three policies: To be more specific, you may say "There are three policies currently being supported". A policy constitutes a schema (=coding algorithm + data unit and parity unit) and a cell size. > HDFS Erasure Coding doc should state its currently supported erasure coding > policies > > > Key: HDFS-10910 > URL: https://issues.apache.org/jira/browse/HDFS-10910 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Yiqun Lin > Attachments: HDFS-10910.001.patch, HDFS-10910.002.patch > > > While HDFS Erasure Coding doc states a variety of possible combinations of > algorithms, block group size and cell size, the code (as of 3.0.0-alpha1) > allows only three policies: RS_6_3_SCHEMA, RS_3_2_SCHEMA and > RS_6_3_LEGACY_SCHEMA. All with default cell size. I think this should be > documented. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10910) HDFS Erasure Coding doc should state its currently supported erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533383#comment-15533383 ] Wei-Chiu Chuang commented on HDFS-10910: Thanks a lot [~linyiqun] for submitting the patch and thanks [~Sammi] for the first review. I think the patch is mostly good, just one nit for clarity: bq. The current codec algorithms support three policies: To be more specific, you may say "There are three policies currently being supported". A policy constitutes a schema (=coding algorithm + data unit and parity unit) and a cell size. > HDFS Erasure Coding doc should state its currently supported erasure coding > policies > > > Key: HDFS-10910 > URL: https://issues.apache.org/jira/browse/HDFS-10910 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Yiqun Lin > Attachments: HDFS-10910.001.patch, HDFS-10910.002.patch > > > While HDFS Erasure Coding doc states a variety of possible combinations of > algorithms, block group size and cell size, the code (as of 3.0.0-alpha1) > allows only three policies: RS_6_3_SCHEMA, RS_3_2_SCHEMA and > RS_6_3_LEGACY_SCHEMA. All with default cell size. I think this should be > documented. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10897) Ozone: SCM: Add NodeManager
[ https://issues.apache.org/jira/browse/HDFS-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533370#comment-15533370 ] Anu Engineer commented on HDFS-10897: - [~jingzhao] I have filed HDFS-10928 & HDFS-10929 to track your comments. > Ozone: SCM: Add NodeManager > --- > > Key: HDFS-10897 > URL: https://issues.apache.org/jira/browse/HDFS-10897 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-10897-HDFS-7240.001.patch, > HDFS-10897-HDFS-7240.002.patch, HDFS-10897-HDFS-7240.003.patch > > > Add a nodeManager class that will be used by Storage Controller Manager > eventually. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10929) Ozone:SCM: explore if we need 3 maps for tracking the state of nodes
Anu Engineer created HDFS-10929: --- Summary: Ozone:SCM: explore if we need 3 maps for tracking the state of nodes Key: HDFS-10929 URL: https://issues.apache.org/jira/browse/HDFS-10929 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Anu Engineer Fix For: HDFS-7240 Based on comments from [~jingzhao], This jira tracks if we really need 3 maps in the SCMNodeManager class or if we should collapse them to a single one. The reason why we have 3 maps is to reduce lock contention. We might be able to collapse this into a single map. This JIRA is to track that action item. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10922) Adding additional unit tests for Trash
[ https://issues.apache.org/jira/browse/HDFS-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533327#comment-15533327 ] Xiaoyu Yao commented on HDFS-10922: --- [~cheersyang], Sounds good to me. Thank you! > Adding additional unit tests for Trash > -- > > Key: HDFS-10922 > URL: https://issues.apache.org/jira/browse/HDFS-10922 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Xiaoyu Yao > > This ticket is opened to track adding unit tests for Trash. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10922) Adding additional unit tests for Trash
[ https://issues.apache.org/jira/browse/HDFS-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10922: -- Assignee: Weiwei Yang > Adding additional unit tests for Trash > -- > > Key: HDFS-10922 > URL: https://issues.apache.org/jira/browse/HDFS-10922 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Xiaoyu Yao >Assignee: Weiwei Yang > > This ticket is opened to track adding unit tests for Trash. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
[ https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533325#comment-15533325 ] Rushabh S Shah commented on HDFS-10921: --- IN HDFS-10843, @Before is replaced by @BeforeClass and @After is replaced by @AfterClass. This can cause contamination of the name system if we create same paths in 2 test cases and want to test different things in both tests. One way I can think to fix that is we can call FSNamesystem#clear before/after every test case. bq. do you think it's reasonable to change restartNameNodes() to wait for active by default? Do we really need to restart namenodes ? If we just call FSNamesystem#clear as @Before annotation, then there is no need to restart namenodes. There is one catch in this. This doesn't clear the datanode state. AFAICT this test suite doesn't depend on the datanode state so we should be fine. Just pinging [~xkrogen] and [~shv] if they have better ideas. > TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode > > > Key: HDFS-10921 > URL: https://issues.apache.org/jira/browse/HDFS-10921 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch > > > Test fails intermittently because the NN is still in safe mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10683) Make class Token$PrivateToken private
[
https://issues.apache.org/jira/browse/HDFS-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10683:
--
Status: In Progress (was: Patch Available)
Sure [~jojochuang], I will take a look.
> Make class Token$PrivateToken private
> -
>
> Key: HDFS-10683
> URL: https://issues.apache.org/jira/browse/HDFS-10683
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: fs, ha, security, security_token
> Attachments: HDFS-10683.001.patch, HDFS-10683.002.patch
>
>
> Avoid {{instanceof}} or typecasting of {{Toke.PrivateToken}} by introducing
> an interface method in {{Token}}. Make class {{Toke.PrivateToken}} private.
> Use a factory method instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10928) Ozone:SCM: Support MXBean for SCM and NodeManager
Anu Engineer created HDFS-10928: --- Summary: Ozone:SCM: Support MXBean for SCM and NodeManager Key: HDFS-10928 URL: https://issues.apache.org/jira/browse/HDFS-10928 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: HDFS-7240 Based on comments from [~jingzhao], This JIRA is to track moving getNodes, getNodeCount etc to a MXBean interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10897) Ozone: SCM: Add NodeManager
[ https://issues.apache.org/jira/browse/HDFS-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533310#comment-15533310 ] Anu Engineer commented on HDFS-10897: - [~jingzhao] Thank you for the review and +1. I will commit this shortly. I will file a set of JIRAs so that your comments and concerns are tracked and we can get back to them later. > Ozone: SCM: Add NodeManager > --- > > Key: HDFS-10897 > URL: https://issues.apache.org/jira/browse/HDFS-10897 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-10897-HDFS-7240.001.patch, > HDFS-10897-HDFS-7240.002.patch, HDFS-10897-HDFS-7240.003.patch > > > Add a nodeManager class that will be used by Storage Controller Manager > eventually. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10429) DataStreamer interrupted warning always appears when using CLI upload file
[
https://issues.apache.org/jira/browse/HDFS-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-10429:
---
Affects Version/s: 2.7.3
> DataStreamer interrupted warning always appears when using CLI upload file
> ---
>
> Key: HDFS-10429
> URL: https://issues.apache.org/jira/browse/HDFS-10429
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
>Priority: Minor
> Attachments: HDFS-10429.1.patch
>
>
> Every time I use 'hdfs dfs -put' upload file, this warning is printed:
> {code:java}
> 16/05/18 20:57:56 WARN hdfs.DataStreamer: Caught exception
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1245)
> at java.lang.Thread.join(Thread.java:1319)
> at
> org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:871)
> at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:519)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:696)
> {code}
> The reason is this: originally, DataStreamer::closeResponder always prints a
> warning about InterruptedException; since HDFS-9812,
> DFSOutputStream::closeImpl always forces threads to close, which causes
> InterruptedException.
> A simple fix is to use debug level log instead of warning level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10850) getEZForPath should NOT throw FNF
[ https://issues.apache.org/jira/browse/HDFS-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533208#comment-15533208 ] Kihwal Lee commented on HDFS-10850: --- +1 on revert. We can then reopen HDFS-9433 and decide what to do. > getEZForPath should NOT throw FNF > - > > Key: HDFS-10850 > URL: https://issues.apache.org/jira/browse/HDFS-10850 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Rakesh R >Priority: Blocker > > HDFS-9433 made an incompatible change to the semantics of getEZForPath. It > used to return the EZ of the closest ancestor path. It never threw FNF. A > common use of getEZForPath to determining if a file can be renamed, or must > be copied due to mismatched EZs. Notably, this has broken hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10609) Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications
[
https://issues.apache.org/jira/browse/HDFS-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-10609:
---
Attachment: HDFS-10609.branch-2.7.01.patch
Branch-2.8 and above refactored DFSClient a lot. Attach a new patch for
branch-2.7
> Uncaught InvalidEncryptionKeyException during pipeline recovery may abort
> downstream applications
> -
>
> Key: HDFS-10609
> URL: https://issues.apache.org/jira/browse/HDFS-10609
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 2.6.0
> Environment: CDH5.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10609.001.patch, HDFS-10609.002.patch,
> HDFS-10609.003.patch, HDFS-10609.004.patch, HDFS-10609.005.patch,
> HDFS-10609.branch-2.7.01.patch
>
>
> In normal operations, if SASL negotiation fails due to
> {{InvalidEncryptionKeyException}}, it is typically a benign exception, which
> is caught and retried :
> {code:title=SaslDataTransferServer#doSaslHandshake}
> if (ioe instanceof SaslException &&
> ioe.getCause() != null &&
> ioe.getCause() instanceof InvalidEncryptionKeyException) {
> // This could just be because the client is long-lived and hasn't gotten
> // a new encryption key from the NN in a while. Upon receiving this
> // error, the client will get a new encryption key from the NN and retry
> // connecting to this DN.
> sendInvalidKeySaslErrorMessage(out, ioe.getCause().getMessage());
> }
> {code}
> {code:title=DFSOutputStream.DataStreamer#createBlockOutputStream}
> if (ie instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) {
> DFSClient.LOG.info("Will fetch a new encryption key and retry, "
> + "encryption key was invalid when connecting to "
> + nodes[0] + " : " + ie);
> {code}
> However, if the exception is thrown during pipeline recovery, the
> corresponding code does not handle it properly, and the exception is spilled
> out to downstream applications, such as SOLR, aborting its operation:
> {quote}
> 2016-07-06 12:12:51,992 ERROR org.apache.solr.update.HdfsTransactionLog:
> Exception closing tlog.
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:474)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:1308)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1272)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1433)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1147)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:632)
> 2016-07-06 12:12:51,997 ERROR org.apache.solr.update.CommitTracker: auto
> commit error...:org.apache.solr.common.SolrException:
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at
> org.apache.solr.update.HdfsTransactionLog.close(HdfsTransactionLog.java:316)
> at
> org.apache.solr.update.TransactionLog.decref(TransactionLog.java:505)
> at org.apache.solr.update.UpdateLog.addOldLog(UpdateLog.java:380)
> at org.apache.solr.update.UpdateLog.postCommit(UpdateLog.java:676)
> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:623)
> at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> at
> java.util.conc
[jira] [Updated] (HDFS-10609) Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications
[
https://issues.apache.org/jira/browse/HDFS-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-10609:
---
Status: Patch Available (was: Reopened)
> Uncaught InvalidEncryptionKeyException during pipeline recovery may abort
> downstream applications
> -
>
> Key: HDFS-10609
> URL: https://issues.apache.org/jira/browse/HDFS-10609
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 2.6.0
> Environment: CDH5.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10609.001.patch, HDFS-10609.002.patch,
> HDFS-10609.003.patch, HDFS-10609.004.patch, HDFS-10609.005.patch,
> HDFS-10609.branch-2.7.01.patch
>
>
> In normal operations, if SASL negotiation fails due to
> {{InvalidEncryptionKeyException}}, it is typically a benign exception, which
> is caught and retried :
> {code:title=SaslDataTransferServer#doSaslHandshake}
> if (ioe instanceof SaslException &&
> ioe.getCause() != null &&
> ioe.getCause() instanceof InvalidEncryptionKeyException) {
> // This could just be because the client is long-lived and hasn't gotten
> // a new encryption key from the NN in a while. Upon receiving this
> // error, the client will get a new encryption key from the NN and retry
> // connecting to this DN.
> sendInvalidKeySaslErrorMessage(out, ioe.getCause().getMessage());
> }
> {code}
> {code:title=DFSOutputStream.DataStreamer#createBlockOutputStream}
> if (ie instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) {
> DFSClient.LOG.info("Will fetch a new encryption key and retry, "
> + "encryption key was invalid when connecting to "
> + nodes[0] + " : " + ie);
> {code}
> However, if the exception is thrown during pipeline recovery, the
> corresponding code does not handle it properly, and the exception is spilled
> out to downstream applications, such as SOLR, aborting its operation:
> {quote}
> 2016-07-06 12:12:51,992 ERROR org.apache.solr.update.HdfsTransactionLog:
> Exception closing tlog.
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:474)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:1308)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1272)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1433)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1147)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:632)
> 2016-07-06 12:12:51,997 ERROR org.apache.solr.update.CommitTracker: auto
> commit error...:org.apache.solr.common.SolrException:
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at
> org.apache.solr.update.HdfsTransactionLog.close(HdfsTransactionLog.java:316)
> at
> org.apache.solr.update.TransactionLog.decref(TransactionLog.java:505)
> at org.apache.solr.update.UpdateLog.addOldLog(UpdateLog.java:380)
> at org.apache.solr.update.UpdateLog.postCommit(UpdateLog.java:676)
> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:623)
> at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concur
[jira] [Reopened] (HDFS-10609) Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications
[
https://issues.apache.org/jira/browse/HDFS-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang reopened HDFS-10609:
Reopen to attach a branch-2.7 patch.
> Uncaught InvalidEncryptionKeyException during pipeline recovery may abort
> downstream applications
> -
>
> Key: HDFS-10609
> URL: https://issues.apache.org/jira/browse/HDFS-10609
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 2.6.0
> Environment: CDH5.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10609.001.patch, HDFS-10609.002.patch,
> HDFS-10609.003.patch, HDFS-10609.004.patch, HDFS-10609.005.patch
>
>
> In normal operations, if SASL negotiation fails due to
> {{InvalidEncryptionKeyException}}, it is typically a benign exception, which
> is caught and retried :
> {code:title=SaslDataTransferServer#doSaslHandshake}
> if (ioe instanceof SaslException &&
> ioe.getCause() != null &&
> ioe.getCause() instanceof InvalidEncryptionKeyException) {
> // This could just be because the client is long-lived and hasn't gotten
> // a new encryption key from the NN in a while. Upon receiving this
> // error, the client will get a new encryption key from the NN and retry
> // connecting to this DN.
> sendInvalidKeySaslErrorMessage(out, ioe.getCause().getMessage());
> }
> {code}
> {code:title=DFSOutputStream.DataStreamer#createBlockOutputStream}
> if (ie instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) {
> DFSClient.LOG.info("Will fetch a new encryption key and retry, "
> + "encryption key was invalid when connecting to "
> + nodes[0] + " : " + ie);
> {code}
> However, if the exception is thrown during pipeline recovery, the
> corresponding code does not handle it properly, and the exception is spilled
> out to downstream applications, such as SOLR, aborting its operation:
> {quote}
> 2016-07-06 12:12:51,992 ERROR org.apache.solr.update.HdfsTransactionLog:
> Exception closing tlog.
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:474)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:1308)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1272)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1433)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1147)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:632)
> 2016-07-06 12:12:51,997 ERROR org.apache.solr.update.CommitTracker: auto
> commit error...:org.apache.solr.common.SolrException:
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at
> org.apache.solr.update.HdfsTransactionLog.close(HdfsTransactionLog.java:316)
> at
> org.apache.solr.update.TransactionLog.decref(TransactionLog.java:505)
> at org.apache.solr.update.UpdateLog.addOldLog(UpdateLog.java:380)
> at org.apache.solr.update.UpdateLog.postCommit(UpdateLog.java:676)
> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:623)
> at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[jira] [Commented] (HDFS-10850) getEZForPath should NOT throw FNF
[ https://issues.apache.org/jira/browse/HDFS-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533129#comment-15533129 ] Daryn Sharp commented on HDFS-10850: I'm not sure querying the EZ for the parent directory is the right fix, and we should consider that code other than hive depends on the old semantics. Depending on which rename is called, and if the dest is a dir, it will move the src _into_ the dest dir. So let's say I want to rename from /dir1/file to /dir2/dir-with-EZ. The result may be /dir2/dir-with-EZ/file. If the parent is queried, ie. /dir2, the rename will look like it should succeed (src and dest have no EZ) but will fail. The old debatably broken semantics cover this case. I realized another similar use case to consider. Let's I want to create a file only if the path is under an EZ. Should I have to query the parent's EZ, catch FNF, repeat until I find an EZ or hit the root dir? The no-FNF semantics require 1 RPC. In the end we may need to consider preserving the no-FNF semantics and add specific exceptions. > getEZForPath should NOT throw FNF > - > > Key: HDFS-10850 > URL: https://issues.apache.org/jira/browse/HDFS-10850 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Rakesh R >Priority: Blocker > > HDFS-9433 made an incompatible change to the semantics of getEZForPath. It > used to return the EZ of the closest ancestor path. It never threw FNF. A > common use of getEZForPath to determining if a file can be renamed, or must > be copied due to mismatched EZs. Notably, this has broken hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java
[ https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533068#comment-15533068 ] Fenghua Hu commented on HDFS-10690: --- [~stack] thanks for reviewing the patch! > Optimize insertion/removal of replica in ShortCircuitCache.java > --- > > Key: HDFS-10690 > URL: https://issues.apache.org/jira/browse/HDFS-10690 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0-alpha2 >Reporter: Fenghua Hu >Assignee: Fenghua Hu > Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, > HDFS-10690.003.patch, HDFS-10690.004.patch, HDFS-10690.005.patch, > HDFS-10690.006.patch, ShortCircuitCache_LinkedMap.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > Currently in ShortCircuitCache, two TreeMap objects are used to track the > cached replicas. > private final TreeMap evictable = new TreeMap<>(); > private final TreeMap evictableMmapped = new > TreeMap<>(); > TreeMap employs Red-Black tree for sorting. This isn't an issue when using > traditional HDD. But when using high-performance SSD/PCIe Flash, the cost > inserting/removing an entry becomes considerable. > To mitigate it, we designed a new list-based for replica tracking. > The list is a double-linked FIFO. FIFO is time-based, thus insertion is a > very low cost operation. On the other hand, list is not lookup-friendly. To > address this issue, we introduce two references into ShortCircuitReplica > object. > ShortCircuitReplica next = null; > ShortCircuitReplica prev = null; > In this way, lookup is not needed when removing a replica from the list. We > only need to modify its predecessor's and successor's references in the lists. > Our tests showed up to 15-50% performance improvement when using PCIe flash > as storage media. > The original patch is against 2.6.4, now I am porting to Hadoop trunk, and > patch will be posted soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
