[jira] [Commented] (HDFS-8527) OzoneHandler: Integration of REST interface and container data pipeline back-end
[ https://issues.apache.org/jira/browse/HDFS-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020316#comment-15020316 ] Anu Engineer commented on HDFS-8527: [~cnauroth] I just merged trunk to HDFS-7240. I will try to get jenkins to re-run this patch. > OzoneHandler: Integration of REST interface and container data pipeline > back-end > > > Key: HDFS-8527 > URL: https://issues.apache.org/jira/browse/HDFS-8527 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-8527-HDFS-7240.001.patch > > > This issue tracks development of OzoneHandler. This is a component within > the DataNode that receives inbound requests parsed from the REST interface, > dispatches to the underlying storage container data pipeline, and then > returns an appropriate response to the REST layer for translation to an > outbound HTTP response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-9289: Attachment: HDFS-9289-branch-2.6.patch Thanks [~sjlee0]. I'm attaching a branch-2.6 patch. Am I updating CHANGES.TXT correctly? If looks good to you I'll push to branch-2.6. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 3.0.0, 2.7.3 > > Attachments: HDFS-9289-branch-2.6.patch, HDFS-9289.1.patch, > HDFS-9289.2.patch, HDFS-9289.3.patch, HDFS-9289.4.patch, HDFS-9289.5.patch, > HDFS-9289.6.patch, HDFS-9289.7.patch, HDFS-9289.branch-2.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8527) OzoneHandler: Integration of REST interface and container data pipeline back-end
[ https://issues.apache.org/jira/browse/HDFS-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020306#comment-15020306 ] Anu Engineer commented on HDFS-8527: No, Please go ahead and merge the trunk. Just so that you know we might have to ignore some failures in ozone due to some EC changes > OzoneHandler: Integration of REST interface and container data pipeline > back-end > > > Key: HDFS-8527 > URL: https://issues.apache.org/jira/browse/HDFS-8527 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-8527-HDFS-7240.001.patch > > > This issue tracks development of OzoneHandler. This is a component within > the DataNode that receives inbound requests parsed from the REST interface, > dispatches to the underlying storage container data pipeline, and then > returns an appropriate response to the REST layer for translation to an > outbound HTTP response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9365) Balaner does not work with the HDFS-6376 HA setup
[
https://issues.apache.org/jira/browse/HDFS-9365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020282#comment-15020282
]
Hadoop QA commented on HDFS-9365:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 23s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 50s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 59s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 29s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 240m 2s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.server.datanode.TestBlockReplacement |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| JDK v1.7.0_85 Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.TestLocalDFS |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
[jira] [Commented] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020262#comment-15020262
]
Mingliang Liu commented on HDFS-9436:
-
Failing tests and ASF license warnings are not related to this change.
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
>
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9436.000.patch
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020242#comment-15020242
]
Hadoop QA commented on HDFS-9436:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 54s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 22s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 54m 50s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 20s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 142m 6s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.server.namenode.TestBackupNode |
| JDK v1.7.0_85 Failed junit tests | hadoop.hdfs.TestDFSClientRetries |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12773631/HDFS-9436.000.patch |
| JIRA Issue | HDFS-9436 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c252138eabf6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ma
[jira] [Commented] (HDFS-9442) Move block replication logic from BlockManager to a new class ReplicationManager
[
https://issues.apache.org/jira/browse/HDFS-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020225#comment-15020225
]
Hadoop QA commented on HDFS-9442:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
3s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 3s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 57s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66
generated 3 new issues (was 33, now 33). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 48s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85 with JDK v1.7.0_85
generated 3 new issues (was 35, now 35). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s
{color} | {color:red} Patch generated 16 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 348, now 320). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 29s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs introduced 1 new FindBugs
issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 2s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 33s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 166m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Unchecked/unconfirmed cast from org.apache.hadoop.hdfs.protocol.Block to
org.apa
[jira] [Commented] (HDFS-9442) Move block replication logic from BlockManager to a new class ReplicationManager
[
https://issues.apache.org/jira/browse/HDFS-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020220#comment-15020220
]
Hadoop QA commented on HDFS-9442:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 8s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66
generated 3 new issues (was 33, now 33). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 49s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85 with JDK v1.7.0_85
generated 3 new issues (was 35, now 35). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s
{color} | {color:red} Patch generated 16 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 348, now 320). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 5s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs introduced 1 new FindBugs
issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 45s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 51m 5s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 49s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 21s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 128m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Unchecked/unconfirmed cast from org.apache.hadoop.hdfs.protocol.Block to
org.apac
[jira] [Commented] (HDFS-6101) TestReplaceDatanodeOnFailure fails occasionally
[ https://issues.apache.org/jira/browse/HDFS-6101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020193#comment-15020193 ] Wei-Chiu Chuang commented on HDFS-6101: --- Thanks for the suggestion! That should resolve the issue. Let me try that later. > TestReplaceDatanodeOnFailure fails occasionally > --- > > Key: HDFS-6101 > URL: https://issues.apache.org/jira/browse/HDFS-6101 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Arpit Agarwal >Assignee: Wei-Chiu Chuang > Attachments: HDFS-6101.001.patch, HDFS-6101.002.patch, > HDFS-6101.003.patch, HDFS-6101.004.patch, HDFS-6101.005.patch, > TestReplaceDatanodeOnFailure.log > > > Exception details in a comment below. > The failure repros on both OS X and Linux if I run the test ~10 times in a > loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9365) Balaner does not work with the HDFS-6376 HA setup
[ https://issues.apache.org/jira/browse/HDFS-9365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-9365: -- Attachment: h9365_20151120.patch > Balaner does not work with the HDFS-6376 HA setup > - > > Key: HDFS-9365 > URL: https://issues.apache.org/jira/browse/HDFS-9365 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h9365_20151119.patch, h9365_20151120.patch > > > HDFS-6376 added support for DistCp between two HA clusters. After the > change, Balaner will use all the NN from both the local and the remote > clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9365) Balaner does not work with the HDFS-6376 HA setup
[ https://issues.apache.org/jira/browse/HDFS-9365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020189#comment-15020189 ] Tsz Wo Nicholas Sze commented on HDFS-9365: --- h9365_20151120.patch: addresses Rakesh's comment. > Balaner does not work with the HDFS-6376 HA setup > - > > Key: HDFS-9365 > URL: https://issues.apache.org/jira/browse/HDFS-9365 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h9365_20151119.patch, h9365_20151120.patch > > > HDFS-6376 added support for DistCp between two HA clusters. After the > change, Balaner will use all the NN from both the local and the remote > clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6101) TestReplaceDatanodeOnFailure fails occasionally
[
https://issues.apache.org/jira/browse/HDFS-6101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020188#comment-15020188
]
Walter Su commented on HDFS-6101:
-
bq. When there are 10 writers begin to writer at the same time, the policy
will not allow some writers set up pipelines with 3 data nodes, due to the load
factor of data nodes.
It happens because in test we only start few DNs and write a lot files. In
production It won't be a problem. I saw nodes be excluded by placement policy
many times when I write tests for erasue-coded files, which writes to 9 DNs
concurrently only for one file.
So could you try
{{conf.setBoolean(DFSConfigKeys.DFS_NAMENODE_REPLICATION_CONSIDERLOAD_KEY,
false);}}
and don't reduce the writers?
> TestReplaceDatanodeOnFailure fails occasionally
> ---
>
> Key: HDFS-6101
> URL: https://issues.apache.org/jira/browse/HDFS-6101
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Arpit Agarwal
>Assignee: Wei-Chiu Chuang
> Attachments: HDFS-6101.001.patch, HDFS-6101.002.patch,
> HDFS-6101.003.patch, HDFS-6101.004.patch, HDFS-6101.005.patch,
> TestReplaceDatanodeOnFailure.log
>
>
> Exception details in a comment below.
> The failure repros on both OS X and Linux if I run the test ~10 times in a
> loop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9314) Improve BlockPlacementPolicyDefault's picking of excess replicas
[
https://issues.apache.org/jira/browse/HDFS-9314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15020186#comment-15020186
]
Ming Ma commented on HDFS-9314:
---
Thanks.
* If there are two replicas node1(rack 1) and node2(rack 2),
{{pickupReplicaSet}} should still return both nodes. Maybe it is useful to have
a new test for this special case.
* Nit: Comment {{storages ({storages[3]}, {storages[7], storages[8]}) are on
different racks (r1, r2, r3)}} should be {{storages ({storages[3]},
{storages[7], storages[8]}) are on different racks (r2, r3, r3)}}.
> Improve BlockPlacementPolicyDefault's picking of excess replicas
>
>
> Key: HDFS-9314
> URL: https://issues.apache.org/jira/browse/HDFS-9314
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Xiao Chen
> Attachments: HDFS-9314.001.patch, HDFS-9314.002.patch,
> HDFS-9314.003.patch, HDFS-9314.004.patch
>
>
> The test case used in HDFS-9313 identified NullPointerException as well as
> the limitation of excess replica picking. If the current replicas are on
> {SSD(rack r1), DISK(rack 2), DISK(rack 3), DISK(rack 3)} and the storage
> policy changes to HOT_STORAGE_POLICY_ID, BlockPlacementPolicyDefault's won't
> be able to delete SSD replica.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4
[ https://issues.apache.org/jira/browse/HDFS-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019290#comment-15019290 ] Joep Rottinghuis commented on HDFS-8791: Probably not-surprisingly I'm a +1 (non-binding) for the patch and to thanking [~ctrezzo] for his work to verify, measure and write up the findings for this. Not to pile on, but here is another mechanism by which heavy disk IO can impact a JVM: http://www.evanjones.ca/jvm-mmap-pause.html (disclaimer: we have not observed this exact interaction with the JVM writing hsperfdata and the disk layout in the wild, I just want to point out the possible connection). > block ID-based DN storage layout can be very slow for datanode on ext4 > -- > > Key: HDFS-8791 > URL: https://issues.apache.org/jira/browse/HDFS-8791 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0, 2.8.0, 2.7.1 >Reporter: Nathan Roberts >Assignee: Chris Trezzo >Priority: Critical > Attachments: 32x32DatanodeLayoutTesting-v1.pdf, > HDFS-8791-trunk-v1.patch > > > We are seeing cases where the new directory layout causes the datanode to > basically cause the disks to seek for 10s of minutes. This can be when the > datanode is running du, and it can also be when it is performing a > checkDirs(). Both of these operations currently scan all directories in the > block pool and that's very expensive in the new layout. > The new layout creates 256 subdirs, each with 256 subdirs. Essentially 64K > leaf directories where block files are placed. > So, what we have on disk is: > - 256 inodes for the first level directories > - 256 directory blocks for the first level directories > - 256*256 inodes for the second level directories > - 256*256 directory blocks for the second level directories > - Then the inodes and blocks to store the the HDFS blocks themselves. > The main problem is the 256*256 directory blocks. > inodes and dentries will be cached by linux and one can configure how likely > the system is to prune those entries (vfs_cache_pressure). However, ext4 > relies on the buffer cache to cache the directory blocks and I'm not aware of > any way to tell linux to favor buffer cache pages (even if it did I'm not > sure I would want it to in general). > Also, ext4 tries hard to spread directories evenly across the entire volume, > this basically means the 64K directory blocks are probably randomly spread > across the entire disk. A du type scan will look at directories one at a > time, so the ioscheduler can't optimize the corresponding seeks, meaning the > seeks will be random and far. > In a system I was using to diagnose this, I had 60K blocks. A DU when things > are hot is less than 1 second. When things are cold, about 20 minutes. > How do things get cold? > - A large set of tasks run on the node. This pushes almost all of the buffer > cache out, causing the next DU to hit this situation. We are seeing cases > where a large job can cause a seek storm across the entire cluster. > Why didn't the previous layout see this? > - It might have but it wasn't nearly as pronounced. The previous layout would > be a few hundred directory blocks. Even when completely cold, these would > only take a few a hundred seeks which would mean single digit seconds. > - With only a few hundred directories, the odds of the directory blocks > getting modified is quite high, this keeps those blocks hot and much less > likely to be evicted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9436:
Status: Patch Available (was: Open)
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
>
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9436.000.patch
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9436:
Attachment: HDFS-9436.000.patch
The v0 patch makes number of datanodes in {{BlockReportStat}} 10 by default.
This is only to test more cases, especially brought by [HDFS-9379].
The default value should be rarely used. Meanwhile, the {{nameNodeProto}} and
{{dataNodeProto}} are static members of the benchmark, thus running multiple
operation tests with different arguments (e.g. number of datanodes) may get
unexpected result. Because of this, the default number of datanodes in
{{ReplicationReportStat}} is also changed as 10.
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
>
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9436.000.patch
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode
[
https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019286#comment-15019286
]
Haohui Mai commented on HDFS-7060:
--
Hi [~nroberts], we're being conservative as you never know whether it will lead
to new race conditions. Can you please test it in your cluster?
> Avoid taking locks when sending heartbeats from the DataNode
>
>
> Key: HDFS-7060
> URL: https://issues.apache.org/jira/browse/HDFS-7060
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Xinwei Qin
> Labels: BB2015-05-TBR
> Attachments: HDFS-7060-002.patch, HDFS-7060.000.patch,
> HDFS-7060.001.patch
>
>
> We're seeing the heartbeat is blocked by the monitor of {{FsDatasetImpl}}
> when the DN is under heavy load of writes:
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getDfsUsed(FsVolumeImpl.java:115)
> - waiting to lock <0x000780304fb8> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getStorageReports(FsDatasetImpl.java:91)
> - locked <0x000780612fd8> (a java.lang.Object)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:563)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:668)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:827)
> at java.lang.Thread.run(Thread.java:744)
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:743)
> - waiting to lock <0x000780304fb8> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:60)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:169)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:621)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:232)
> at java.lang.Thread.run(Thread.java:744)
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1006)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:59)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:244)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:195)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:753)
> - locked <0x000780304fb8> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:60)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:169)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:621)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:232)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9416) Respect OpenSSL and protobuf definitions in maven configuration when building libhdfspp
[ https://issues.apache.org/jira/browse/HDFS-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019277#comment-15019277 ] Allen Wittenauer commented on HDFS-9416: bq. Allen Wittenauer would you please change mvn command line by passing them? In a nutshell, no. The Dockerfile used by Jenkins installs these packages in the place where most Linux systems install those components. There should be zero reason to specify those locations. The directives given via Maven are to be used specifically in the case where the normal locations that are used during autodetection are broken in some way. For example, the OpenSSL implementation used in OS X Mavericks doesn't support a feature that libhadoop.so expects to be able to use. Therefore, many people use homebrew to install it into /usr/local. > Respect OpenSSL and protobuf definitions in maven configuration when building > libhdfspp > --- > > Key: HDFS-9416 > URL: https://issues.apache.org/jira/browse/HDFS-9416 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Haohui Mai >Assignee: Xiaobing Zhou >Priority: Blocker > Attachments: HDFS-9416.004.patch, HDFS-9416.HDFS-8707.004.patch, > HDFS-9416.HDFS-8707.005.patch > > > As discovered in HDFS-9380 the current pom.xml / CMakeLists.txt in libhdfspp > does not respect the configuration from the maven command line. Subsequently > it breaks the Jenkins build. > Both pom.xml and CMakeLists.txt need to be fixed to get Jenkins working again. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9442) Move block replication logic from BlockManager to a new class ReplicationManager
[
https://issues.apache.org/jira/browse/HDFS-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9442:
Attachment: HDFS-9442.001.patch
> Move block replication logic from BlockManager to a new class
> ReplicationManager
>
>
> Key: HDFS-9442
> URL: https://issues.apache.org/jira/browse/HDFS-9442
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-9442.000.patch, HDFS-9442.001.patch
>
>
> Currently the {{BlockManager}} is managing all replication logic for over- ,
> under- and mis-replicated blocks. This jira proposes to move that code to a
> new class named {{ReplicationManager}} for cleaner code logic, shorter source
> files, and easier lock separating work in future.
> The {{ReplicationManager}} is a package local class, providing
> {{BlockManager}} with methods that accesses its internal data structures of
> replication queue. Meanwhile, the class maintains the lifecycle of
> {{replicationThread}} and {{replicationQueuesInitializer}} daemon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9442) Move block replication logic from BlockManager to a new class ReplicationManager
[
https://issues.apache.org/jira/browse/HDFS-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9442:
Attachment: (was: HDFS-9442.001.patch)
> Move block replication logic from BlockManager to a new class
> ReplicationManager
>
>
> Key: HDFS-9442
> URL: https://issues.apache.org/jira/browse/HDFS-9442
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-9442.000.patch, HDFS-9442.001.patch
>
>
> Currently the {{BlockManager}} is managing all replication logic for over- ,
> under- and mis-replicated blocks. This jira proposes to move that code to a
> new class named {{ReplicationManager}} for cleaner code logic, shorter source
> files, and easier lock separating work in future.
> The {{ReplicationManager}} is a package local class, providing
> {{BlockManager}} with methods that accesses its internal data structures of
> replication queue. Meanwhile, the class maintains the lifecycle of
> {{replicationThread}} and {{replicationQueuesInitializer}} daemon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019228#comment-15019228 ] Sangjin Lee commented on HDFS-9289: --- Thanks [~zhz]! Could you please cherry-pick the commit to branch-2.6? > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 3.0.0, 2.7.3 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.7.patch, HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019220#comment-15019220 ] Zhe Zhang commented on HDFS-9289: - I think this is a good addition to branch-2.6. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 3.0.0, 2.7.3 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.7.patch, HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019222#comment-15019222 ] Zhe Zhang commented on HDFS-9289: - [~sjlee0] Yes this is a valid bug even for 2.6.x. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 3.0.0, 2.7.3 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.7.patch, HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9348) Erasure Coding: DFS GetErasureCodingPolicy API on a non-existent file should be handled properly
[
https://issues.apache.org/jira/browse/HDFS-9348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019208#comment-15019208
]
Uma Maheswara Rao G commented on HDFS-9348:
---
+1 on the latest patch. I will commit it shortly.
> Erasure Coding: DFS GetErasureCodingPolicy API on a non-existent file should
> be handled properly
>
>
> Key: HDFS-9348
> URL: https://issues.apache.org/jira/browse/HDFS-9348
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Minor
> Attachments: HDFS-9348-00.patch, HDFS-9348-002.patch,
> HDFS-9348.branch-2.00.patch
>
>
> Presently calling {{dfs#getErasureCodingPolicy()}} on a non-existent file is
> returning the ErasureCodingPolicy info. As per the
> [discussion|https://issues.apache.org/jira/browse/HDFS-8777?focusedCommentId=14981077&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14981077]
> it has to validate and throw FileNotFoundException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9442) Move block replication logic from BlockManager to a new class ReplicationManager
[
https://issues.apache.org/jira/browse/HDFS-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9442:
Attachment: HDFS-9442.001.patch
The v1 patch moves the {{neededReplication}} from {{BlockManager}} to
{{ReplicationManager}} class.
> Move block replication logic from BlockManager to a new class
> ReplicationManager
>
>
> Key: HDFS-9442
> URL: https://issues.apache.org/jira/browse/HDFS-9442
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-9442.000.patch, HDFS-9442.001.patch
>
>
> Currently the {{BlockManager}} is managing all replication logic for over- ,
> under- and mis-replicated blocks. This jira proposes to move that code to a
> new class named {{ReplicationManager}} for cleaner code logic, shorter source
> files, and easier lock separating work in future.
> The {{ReplicationManager}} is a package local class, providing
> {{BlockManager}} with methods that accesses its internal data structures of
> replication queue. Meanwhile, the class maintains the lifecycle of
> {{replicationThread}} and {{replicationQueuesInitializer}} daemon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9416) Respect OpenSSL and protobuf definitions in maven configuration when building libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019173#comment-15019173
]
Xiaobing Zhou commented on HDFS-9416:
-
I looked into FindProtobuf.cmake source in cmake, it doesn't do protobuf
include-dir/lib auto search.
[~aw] would you please change mvn command line by passing them? .e.g,
{code}
-Dprotobuf.lib=/root/install/lib/libprotobuf.so
-Dprotobuf.include.dir=/root/install/include/
-Dprotoc.lib=/root/install/lib/libprotoc.so
-Dprotoc.exec=/root/install/bin/protoc
{code}
> Respect OpenSSL and protobuf definitions in maven configuration when building
> libhdfspp
> ---
>
> Key: HDFS-9416
> URL: https://issues.apache.org/jira/browse/HDFS-9416
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Haohui Mai
>Assignee: Xiaobing Zhou
>Priority: Blocker
> Attachments: HDFS-9416.004.patch, HDFS-9416.HDFS-8707.004.patch,
> HDFS-9416.HDFS-8707.005.patch
>
>
> As discovered in HDFS-9380 the current pom.xml / CMakeLists.txt in libhdfspp
> does not respect the configuration from the maven command line. Subsequently
> it breaks the Jenkins build.
> Both pom.xml and CMakeLists.txt need to be fixed to get Jenkins working again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9443) Disabling HDFS client socket cache causes logging message printed to console for CLI commands.
[
https://issues.apache.org/jira/browse/HDFS-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019150#comment-15019150
]
Hadoop QA commented on HDFS-9443:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
2s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 8s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 6s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 6s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Issue | HDFS-9443 |
| GITHUB PR | https://github.com/apache/hadoop/pull/49 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8af31b876afc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk /
[jira] [Commented] (HDFS-6945) BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed
[ https://issues.apache.org/jira/browse/HDFS-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019129#comment-15019129 ] Sangjin Lee commented on HDFS-6945: --- Does this issue exist in 2.6.x? Should this be backported to branch-2.6? > BlockManager should remove a block from excessReplicateMap and decrement > ExcessBlocks metric when the block is removed > -- > > Key: HDFS-6945 > URL: https://issues.apache.org/jira/browse/HDFS-6945 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.5.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA >Priority: Critical > Labels: metrics > Fix For: 2.8.0, 2.7.2 > > Attachments: HDFS-6945-003.patch, HDFS-6945-004.patch, > HDFS-6945-005.patch, HDFS-6945.2.patch, HDFS-6945.patch > > > I'm seeing ExcessBlocks metric increases to more than 300K in some clusters, > however, there are no over-replicated blocks (confirmed by fsck). > After a further research, I noticed when deleting a block, BlockManager does > not remove the block from excessReplicateMap or decrement excessBlocksCount. > Usually the metric is decremented when processing block report, however, if > the block has been deleted, BlockManager does not remove the block from > excessReplicateMap or decrement the metric. > That way the metric and excessReplicateMap can increase infinitely (i.e. > memory leak can occur). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8676) Delayed rolling upgrade finalization can cause heartbeat expiration and write failures
[ https://issues.apache.org/jira/browse/HDFS-8676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019118#comment-15019118 ] Sangjin Lee commented on HDFS-8676: --- Does this issue exist in 2.6.x? Should this be backported to branch-2.6? > Delayed rolling upgrade finalization can cause heartbeat expiration and write > failures > -- > > Key: HDFS-8676 > URL: https://issues.apache.org/jira/browse/HDFS-8676 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Walter Su >Priority: Critical > Fix For: 3.0.0, 2.7.2 > > Attachments: HDFS-8676.01.patch, HDFS-8676.02.patch > > > In big busy clusters where the deletion rate is also high, a lot of blocks > can pile up in the datanode trash directories until an upgrade is finalized. > When it is finally finalized, the deletion of trash is done in the service > actor thread's context synchronously. This blocks the heartbeat and can > cause heartbeat expiration. > We have seen a namenode losing hundreds of nodes after a delayed upgrade > finalization. The deletion of trash directories should be made asynchronous. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8527) OzoneHandler: Integration of REST interface and container data pipeline back-end
[ https://issues.apache.org/jira/browse/HDFS-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019116#comment-15019116 ] Chris Nauroth commented on HDFS-8527: - [~anu], it looks like we need the HDFS-7240 branch to pick up a recent trunk change in the Dockerfile used by test-patch. Do you have any objection to me doing a merge from trunk to HDFS-7240? It looks like it will be a clean merge. > OzoneHandler: Integration of REST interface and container data pipeline > back-end > > > Key: HDFS-8527 > URL: https://issues.apache.org/jira/browse/HDFS-8527 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-8527-HDFS-7240.001.patch > > > This issue tracks development of OzoneHandler. This is a component within > the DataNode that receives inbound requests parsed from the REST interface, > dispatches to the underlying storage container data pipeline, and then > returns an appropriate response to the REST layer for translation to an > outbound HTTP response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8767) RawLocalFileSystem.listStatus() returns null for UNIX pipefile
[ https://issues.apache.org/jira/browse/HDFS-8767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019113#comment-15019113 ] Sangjin Lee commented on HDFS-8767: --- Does this issue exist in 2.6.x? Should this be backported to branch-2.6? > RawLocalFileSystem.listStatus() returns null for UNIX pipefile > -- > > Key: HDFS-8767 > URL: https://issues.apache.org/jira/browse/HDFS-8767 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Kanaka Kumar Avvaru >Priority: Critical > Fix For: 2.7.2 > > Attachments: HDFS-8767-00.patch, HDFS-8767-01.patch, > HDFS-8767-02.patch, HDFS-8767.003.patch, HDFS-8767.004.patch > > > Calling FileSystem.listStatus() on a UNIX pipe file returns null instead of > the file. The bug breaks Hive when Hive loads data from UNIX pipe file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8995) Flaw in registration bookeeping can make DN die on reconnect
[ https://issues.apache.org/jira/browse/HDFS-8995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019107#comment-15019107 ] Sangjin Lee commented on HDFS-8995: --- Does this issue exist in 2.6.x? Should this be backported to branch-2.6? > Flaw in registration bookeeping can make DN die on reconnect > > > Key: HDFS-8995 > URL: https://issues.apache.org/jira/browse/HDFS-8995 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 2.7.2 > > Attachments: HDFS-8995.patch > > > Normally data nodes re-register with the namenode when it was unreachable for > more than the heartbeat expiration and becomes reachable again. Datanodes > keep retrying the last rpc call such as incremental block report and > heartbeat and when it finally gets through the namenode tells it to > re-register. > We have observed that some of datanodes stay dead in such scenarios. Further > investigation has revealed that those were told to shutdown by the namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8891) HDFS concat should keep srcs order
[ https://issues.apache.org/jira/browse/HDFS-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019108#comment-15019108 ] Sangjin Lee commented on HDFS-8891: --- Does this issue exist in 2.6.x? Should this be backported to branch-2.6? > HDFS concat should keep srcs order > -- > > Key: HDFS-8891 > URL: https://issues.apache.org/jira/browse/HDFS-8891 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yong Zhang >Assignee: Yong Zhang >Priority: Blocker > Fix For: 2.7.2 > > Attachments: HDFS-8891.001.patch, HDFS-8891.002.patch > > > FSDirConcatOp.verifySrcFiles may change src files order, but it should their > order as input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019105#comment-15019105
]
Sangjin Lee commented on HDFS-9083:
---
Should this be backported to branch-2.6?
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2
>
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9441) Do not construct path string when choosing block placement targets
[ https://issues.apache.org/jira/browse/HDFS-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-9441: -- Summary: Do not construct path string when choosing block placement targets (was: Do not call construct path string when choosing block placement targets) > Do not construct path string when choosing block placement targets > -- > > Key: HDFS-9441 > URL: https://issues.apache.org/jira/browse/HDFS-9441 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Attachments: h9441_20151118.patch, h9441_20151119.patch > > > - INodeFile.getName() is expensive since it involves quite a few string > operations. The method is called in both ReplicationWork and > ErasureCodingWork but the default BlockPlacementPolicy does not use the > returned string. We should simply pass BlockCollection to reduce unnecessary > computation when using the default BlockPlacementPolicy. > - Another improvement: the return type of FSNamesystem.getBlockCollection > should be changed to INodeFile since it always returns an INodeFile object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9106) Transfer failure during pipeline recovery causes permanent write failures
[ https://issues.apache.org/jira/browse/HDFS-9106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019098#comment-15019098 ] Sangjin Lee commented on HDFS-9106: --- Does this issue exist in 2.6.x? Should this be backported to branch-2.6? > Transfer failure during pipeline recovery causes permanent write failures > - > > Key: HDFS-9106 > URL: https://issues.apache.org/jira/browse/HDFS-9106 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 2.7.2 > > Attachments: HDFS-9106-poc.patch, HDFS-9106.branch-2.7.patch, > HDFS-9106.patch > > > When a new node is added to a write pipeline during flush/sync, if the > partial block transfer fails, the write will fail permanently without > retrying or continuing with whatever is in the pipeline. > The transfer often fails in busy clusters due to timeout. There is no > per-packet ACK between client and datanode or between source and target > datanodes. If the total transfer time exceeds the configured timeout + 10 > seconds (2 * 5 seconds slack), it is considered failed. Naturally, the > failure rate is higher with bigger block sizes. > I propose following changes: > - Transfer timeout needs to be different from per-packet timeout. > - transfer should be retried if fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad
[
https://issues.apache.org/jira/browse/HDFS-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019094#comment-15019094
]
Sangjin Lee commented on HDFS-9178:
---
Does this issue exist in 2.6.x? Should this be backported to branch-2.6?
> Slow datanode I/O can cause a wrong node to be marked bad
> -
>
> Key: HDFS-9178
> URL: https://issues.apache.org/jira/browse/HDFS-9178
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Critical
> Fix For: 3.0.0, 2.7.2
>
> Attachments: HDFS-9178.branch-2.6.patch, HDFS-9178.patch
>
>
> When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the
> downstream node can timeout on reading packet since even the heartbeat
> packets will not be relayed down.
> The packet read timeout is set in {{DataXceiver#run()}}:
> {code}
> peer.setReadTimeout(dnConf.socketTimeout);
> {code}
> When the downstream node times out and closes the connection to the upstream,
> the upstream node's {{PacketResponder}} gets {{EOFException}} and it sends an
> ack upstream with the downstream node status set to {{ERROR}}. This caused
> the client to exclude the downstream node, even thought the upstream node was
> the one got stuck.
> The connection to downstream has longer timeout, so the downstream will
> always timeout first. The downstream timeout is set in {{writeBlock()}}
> {code}
> int timeoutValue = dnConf.socketTimeout +
> (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length);
> int writeTimeout = dnConf.socketWriteTimeout +
> (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length);
> NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue);
> OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock,
> writeTimeout);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8527) OzoneHandler: Integration of REST interface and container data pipeline back-end
[
https://issues.apache.org/jira/browse/HDFS-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019091#comment-15019091
]
Hadoop QA commented on HDFS-8527:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 15m 25s
{color} | {color:red} Docker failed to build yetus/hadoop:123b3db. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12773614/HDFS-8527-HDFS-7240.001.patch
|
| JIRA Issue | HDFS-8527 |
| Powered by | Apache Yetus http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13588/console |
This message was automatically generated.
> OzoneHandler: Integration of REST interface and container data pipeline
> back-end
>
>
> Key: HDFS-8527
> URL: https://issues.apache.org/jira/browse/HDFS-8527
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-8527-HDFS-7240.001.patch
>
>
> This issue tracks development of OzoneHandler. This is a component within
> the DataNode that receives inbound requests parsed from the REST interface,
> dispatches to the underlying storage container data pipeline, and then
> returns an appropriate response to the REST layer for translation to an
> outbound HTTP response.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019084#comment-15019084 ] Sangjin Lee commented on HDFS-9289: --- Does this issue exist in 2.6.x? Should this be backported to branch-2.6? > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 3.0.0, 2.7.3 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.7.patch, HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8855) Webhdfs client leaks active NameNode connections
[
https://issues.apache.org/jira/browse/HDFS-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019077#comment-15019077
]
Hadoop QA commented on HDFS-8855:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
36s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
41s {color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 53s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 5s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
17s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
14s {color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 1s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 5s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 49s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 11s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 56s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 25s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 23s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 271m 25s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.ipc.TestDecayRpcScheduler |
| | hadoop.fs.shell.find.TestFind |
| | hadoop.test.TestTimedOutTestsListener |
| | hadoop.fs.shell.find.TestPrint |
|
[jira] [Updated] (HDFS-8527) OzoneHandler: Integration of REST interface and container data pipeline back-end
[ https://issues.apache.org/jira/browse/HDFS-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-8527: Status: Patch Available (was: Open) > OzoneHandler: Integration of REST interface and container data pipeline > back-end > > > Key: HDFS-8527 > URL: https://issues.apache.org/jira/browse/HDFS-8527 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-8527-HDFS-7240.001.patch > > > This issue tracks development of OzoneHandler. This is a component within > the DataNode that receives inbound requests parsed from the REST interface, > dispatches to the underlying storage container data pipeline, and then > returns an appropriate response to the REST layer for translation to an > outbound HTTP response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8527) OzoneHandler: Integration of REST interface and container data pipeline back-end
[ https://issues.apache.org/jira/browse/HDFS-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-8527: Attachment: HDFS-8527-HDFS-7240.001.patch I'm attaching a patch that sets up the REST endpoints for Ozone in the DataNode's HTTP server. The REST calls integrate with the back-end container data pipeline handling. Right now, the back-end portion of this is stubbed, just so that we can proceed with exercising the HTTP portion. This took some effort to hoist a custom Jersey container into the Netty pipeline so that we can take advantage of the JAX-RS annotations on the REST service defintion. Long-term, after interfaces settle, I want to consider removing this in favor of coding our own controller, similar to what the existing DataNode HTTP endpoints do. Right now, it's very convenient to be able to use the JAX-RS annotations while the interface evolves. This is also the first time I've tried executing test-patch.sh on the Ozone branch. I took the opportunity to clean up a few things even though some of the changes aren't directly related to this patch. [~anu], would you please review? Thank you! > OzoneHandler: Integration of REST interface and container data pipeline > back-end > > > Key: HDFS-8527 > URL: https://issues.apache.org/jira/browse/HDFS-8527 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-8527-HDFS-7240.001.patch > > > This issue tracks development of OzoneHandler. This is a component within > the DataNode that receives inbound requests parsed from the REST interface, > dispatches to the underlying storage container data pipeline, and then > returns an appropriate response to the REST layer for translation to an > outbound HTTP response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7163) WebHdfsFileSystem should retry reads according to the configured retry policy.
[ https://issues.apache.org/jira/browse/HDFS-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019039#comment-15019039 ] Eric Payne commented on HDFS-7163: -- [~wheat9], thank you for your review and comments on this feature. bq. I think retrying only on the data node is problematic as the retry might have little value when the DN goes down. In this patch, if the DN that is being read from goes down, WebHDFS will put that DN into the client's URL exclude list before querying the NN again for another DN. The only time the same DN is reused is if a seek has occurred. bq. An alternative approach is to have WebHDFS (1) expose a GET_BLOCK call where the DN returns the block directly, and (2) be a smarter client that retries based on block locations. Although this may be a more elegant solution, I think that could be done as part of a separate JIRA, given that we can take advantage of the exclude list functionality as I mentioned above. > WebHdfsFileSystem should retry reads according to the configured retry policy. > -- > > Key: HDFS-7163 > URL: https://issues.apache.org/jira/browse/HDFS-7163 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.5.1 >Reporter: Eric Payne >Assignee: Eric Payne > Attachments: HDFS-7163-branch-2.003.patch, > HDFS-7163-branch-2.7.003.patch, HDFS-7163.001.patch, HDFS-7163.002.patch, > HDFS-7163.003.patch, WebHDFS Read Retry.pdf > > > In the current implementation of WebHdfsFileSystem, opens are retried > according to the configured retry policy, but not reads. Therefore, if a > connection goes down while data is being read, the read will fail and the > read will have to be retried by the client code. > Also, after a connection has been established, the next read (or seek/read) > will fail and the read will have to be restarted by the client code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6101) TestReplaceDatanodeOnFailure fails occasionally
[
https://issues.apache.org/jira/browse/HDFS-6101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019035#comment-15019035
]
Hadoop QA commented on HDFS-6101:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 18s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 52s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 21s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 155m 29s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.TestDFSStripedOutputStream |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
| JDK v1.7.0_85 Failed junit tests | hadoop.hdfs.TestRecoverStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:date2015-11-20 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12773584/HDFS-6101.005.patch |
| JIRA Issue | HDFS-6101 |
| Optional Tests | asflicense compile javac javado
[jira] [Updated] (HDFS-9443) Disabling HDFS client socket cache causes logging message printed to console for CLI commands.
[
https://issues.apache.org/jira/browse/HDFS-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-9443:
Attachment: HDFS-9443.001.patch
> Disabling HDFS client socket cache causes logging message printed to console
> for CLI commands.
> --
>
> Key: HDFS-9443
> URL: https://issues.apache.org/jira/browse/HDFS-9443
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Trivial
> Attachments: HDFS-9443.001.patch
>
>
> The HDFS client's socket cache can be disabled by setting
> {{dfs.client.socketcache.capacity}} to {{0}}. When this is done, the
> {{PeerCache}} class logs an info-level message stating that the cache is
> disabled. This message is getting printed to the console for CLI commands,
> which disrupts CLI output. This issue proposes to downgrade to debug-level
> logging for this message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9443) Disabling HDFS client socket cache causes logging message printed to console for CLI commands.
[
https://issues.apache.org/jira/browse/HDFS-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-9443:
Status: Open (was: Patch Available)
> Disabling HDFS client socket cache causes logging message printed to console
> for CLI commands.
> --
>
> Key: HDFS-9443
> URL: https://issues.apache.org/jira/browse/HDFS-9443
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Trivial
> Attachments: HDFS-9443.001.patch
>
>
> The HDFS client's socket cache can be disabled by setting
> {{dfs.client.socketcache.capacity}} to {{0}}. When this is done, the
> {{PeerCache}} class logs an info-level message stating that the cache is
> disabled. This message is getting printed to the console for CLI commands,
> which disrupts CLI output. This issue proposes to downgrade to debug-level
> logging for this message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9443) Disabling HDFS client socket cache causes logging message printed to console for CLI commands.
[
https://issues.apache.org/jira/browse/HDFS-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-9443:
Status: Patch Available (was: Open)
> Disabling HDFS client socket cache causes logging message printed to console
> for CLI commands.
> --
>
> Key: HDFS-9443
> URL: https://issues.apache.org/jira/browse/HDFS-9443
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Trivial
> Attachments: HDFS-9443.001.patch
>
>
> The HDFS client's socket cache can be disabled by setting
> {{dfs.client.socketcache.capacity}} to {{0}}. When this is done, the
> {{PeerCache}} class logs an info-level message stating that the cache is
> disabled. This message is getting printed to the console for CLI commands,
> which disrupts CLI output. This issue proposes to downgrade to debug-level
> logging for this message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9381) When same block came for replication for Striped mode, we can move that block to PendingReplications
[
https://issues.apache.org/jira/browse/HDFS-9381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019018#comment-15019018
]
Zhe Zhang commented on HDFS-9381:
-
Thanks Uma for the work! I looked over the patch and the approach looks good.
I'll post a full review soon.
Adding to the discussion on how important the optimization is: I think besides
reducing locking contention, this change also speeds up the recovery of
non-striping blocks. E.g., when a rack fails, there could be a lot of striped
block recovery work waiting. They could block regular recovery tasks. And I
think by using another list, we are not slowing down decomm anymore.
> When same block came for replication for Striped mode, we can move that block
> to PendingReplications
>
>
> Key: HDFS-9381
> URL: https://issues.apache.org/jira/browse/HDFS-9381
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding, namenode
>Affects Versions: 3.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-9381.00.patch, HDFS-9381.01.patch
>
>
> Currently I noticed that we are just returning null if block already exists
> in pendingReplications in replication flow for striped blocks.
> {code}
> if (block.isStriped()) {
> if (pendingNum > 0) {
> // Wait the previous recovery to finish.
> return null;
> }
> {code}
> Here if we just return null and if neededReplications contains only fewer
> blocks(basically by default if less than numliveNodes*2), then same blocks
> can be picked again from neededReplications from next loop as we are not
> removing element from neededReplications. Since this replication process need
> to take fsnamesystmem lock and do, we may spend some time unnecessarily in
> every loop.
> So my suggestion/improvement is:
> Instead of just returning null, how about incrementing pendingReplications
> for this block and remove from neededReplications? and also another point to
> consider here is, to add into pendingReplications, generally we need target
> and it is nothing but to which node we issued replication command. Later when
> after replication success and DN reported it, block will be removed from
> pendingReplications from NN addBlock.
> So since this is newly picked block from neededReplications, we would not
> have selected target yet. So which target to be passed to pendingReplications
> if we add this block? One Option I am thinking is, how about just passing
> srcNode itself as target for this special condition? So, anyway if the block
> is really missed, srcNode will not report it. So this block will not be
> removed from pending replications, so that when it is timed out, it will be
> considered for replication again and that time it will find actual target to
> replicate while processing as part of regular replication flow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9314) Improve BlockPlacementPolicyDefault's picking of excess replicas
[
https://issues.apache.org/jira/browse/HDFS-9314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019008#comment-15019008
]
Hadoop QA commented on HDFS-9314:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 58s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s
{color} | {color:red} Patch generated 1 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 49, now 50). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 57s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 58s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 54m 10s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 21s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 141m 27s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.server.namenode.TestBackupNode |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.TestDataTransferKeepalive |
| JDK v1.7.0_85 Failed junit tests | hadoop.fs.TestUnbuffer |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:date2015-11-20 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-9438) TestPipelinesFailover assumes Linux ifconfig
[
https://issues.apache.org/jira/browse/HDFS-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018999#comment-15018999
]
John Zhuge commented on HDFS-9438:
--
{code:java}
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPipelinesFailover.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPipelinesFailover.java
index 3da37f5..1cba792 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPipelinesFailover.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPipelinesFailover.java
@@ -429,6 +429,11 @@ public void testPipelineRecoveryStress() throws Exception {
// The following section of code is to help debug HDFS-6694 about
// this test that fails from time to time due to "too many open files".
//
+if (!Shell.LINUX && !Shell.SOLARIS) {
+System.out.println("Unsupport OS. Skip HDFS-6694 Debug Data");
+return;
+}
+
String[] scmd = new String[] {"/bin/sh", "-c", "ulimit -a"};
ShellCommandExecutor sce = new ShellCommandExecutor(scmd);
sce.execute();
@@ -441,7 +446,7 @@ public void testPipelineRecoveryStress() throws Exception {
sce.execute();
System.out.println("'hostname' output:\n" + sce.getOutput());
-scmd = new String[] {"ifconfig"};
+scmd = new String[] {"ifconfig -a"};
sce = new ShellCommandExecutor(scmd);
sce.execute();
System.out.println("'ifconfig' output:\n" + sce.getOutput());
{code}
> TestPipelinesFailover assumes Linux ifconfig
>
>
> Key: HDFS-9438
> URL: https://issues.apache.org/jira/browse/HDFS-9438
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Affects Versions: 2.7.1
> Environment: Solaris
>Reporter: Alan Burlison
>Assignee: John Zhuge
>Priority: Minor
>
> TestPipelinesFailover.java contains the following:
> {code}
> scmd = new String[] {"ifconfig"};
> sce = new ShellCommandExecutor(scmd);
> sce.execute();
> System.out.println("'ifconfig' output:\n" + sce.getOutput());
> {code}
> That assumes the Linux ifconfig command. If the flag "-a" is added, the same
> invocation should work on both Linux and Solaris - the output is only
> displayed for debugging purposes so the fact that the output of ifconfig is
> different on Linux and Solaris shouldn't matter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9336) deleteSnapshot throws NPE when snapshotname is null
[
https://issues.apache.org/jira/browse/HDFS-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018971#comment-15018971
]
Hadoop QA commented on HDFS-9336:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 15s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s
{color} | {color:red} Patch generated 1 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 57, now 57). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 13s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 5s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 29s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 208m 21s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.server.datanode.TestBlockReplacement |
| | hadoop.fs.TestSymlinkHdfsFileContext |
| | hadoop.hdfs.qjournal.client.TestQuorumJournalManager |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
[jira] [Commented] (HDFS-9038) Reserved space is erroneously counted towards non-DFS used.
[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018962#comment-15018962
]
Hadoop QA commented on HDFS-9038:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 34s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 10s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 2s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 4s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 37s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 28s
{color} | {color:red} hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66
generated 3 new issues (was 49, now 49). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 28s
{color} | {color:red} hadoop-hdfs-project-jdk1.7.0_85 with JDK v1.7.0_85
generated 3 new issues (was 51, now 51). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 28s
{color} | {color:red} Patch generated 4 new checkstyle issues in
hadoop-hdfs-project (total was 292, now 294). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 4s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 24s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 30s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 46s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color
[jira] [Commented] (HDFS-9416) Respect OpenSSL and protobuf definitions in maven configuration when building libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018894#comment-15018894
]
Hadoop QA commented on HDFS-9416:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
9s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 14s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 21s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s
{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 29s
{color} | {color:red} hadoop-hdfs-native-client in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 29s {color} |
{color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 29s {color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 30s
{color} | {color:red} hadoop-hdfs-native-client in the patch failed with JDK
v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 30s {color} |
{color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.7.0_85.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 30s {color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.7.0_85.
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 29s {color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 31s {color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.7.0_85.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 26s
{color} | {color:red} Patch generated 427 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 37s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12773561/HDFS-9416.HDFS-8707.005.patch
|
| JIRA Issue | HDFS-9416 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml cc |
| uname | Linux ba43b1e65987 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_
[jira] [Commented] (HDFS-9438) TestPipelinesFailover assumes Linux ifconfig
[
https://issues.apache.org/jira/browse/HDFS-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018879#comment-15018879
]
Yongjun Zhang commented on HDFS-9438:
-
Thanks [~jzhuge] for looking into this.
> TestPipelinesFailover assumes Linux ifconfig
>
>
> Key: HDFS-9438
> URL: https://issues.apache.org/jira/browse/HDFS-9438
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Affects Versions: 2.7.1
> Environment: Solaris
>Reporter: Alan Burlison
>Assignee: John Zhuge
>Priority: Minor
>
> TestPipelinesFailover.java contains the following:
> {code}
> scmd = new String[] {"ifconfig"};
> sce = new ShellCommandExecutor(scmd);
> sce.execute();
> System.out.println("'ifconfig' output:\n" + sce.getOutput());
> {code}
> That assumes the Linux ifconfig command. If the flag "-a" is added, the same
> invocation should work on both Linux and Solaris - the output is only
> displayed for debugging purposes so the fact that the output of ifconfig is
> different on Linux and Solaris shouldn't matter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-9438) TestPipelinesFailover assumes Linux ifconfig
[
https://issues.apache.org/jira/browse/HDFS-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge reassigned HDFS-9438:
Assignee: John Zhuge (was: Yongjun Zhang)
> TestPipelinesFailover assumes Linux ifconfig
>
>
> Key: HDFS-9438
> URL: https://issues.apache.org/jira/browse/HDFS-9438
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Affects Versions: 2.7.1
> Environment: Solaris
>Reporter: Alan Burlison
>Assignee: John Zhuge
>Priority: Minor
>
> TestPipelinesFailover.java contains the following:
> {code}
> scmd = new String[] {"ifconfig"};
> sce = new ShellCommandExecutor(scmd);
> sce.execute();
> System.out.println("'ifconfig' output:\n" + sce.getOutput());
> {code}
> That assumes the Linux ifconfig command. If the flag "-a" is added, the same
> invocation should work on both Linux and Solaris - the output is only
> displayed for debugging purposes so the fact that the output of ifconfig is
> different on Linux and Solaris shouldn't matter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9426) Rollingupgrade finalization is not backward compatible
[
https://issues.apache.org/jira/browse/HDFS-9426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018872#comment-15018872
]
Hadoop QA commented on HDFS-9426:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 34s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66
generated 3 new issues (was 33, now 33). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 22s {color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79 with JDK v1.7.0_79
generated 3 new issues (was 35, now 35). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s
{color} | {color:red} Patch generated 1 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 15, now 16). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 29s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 26s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 2
[jira] [Comment Edited] (HDFS-9416) Respect OpenSSL and protobuf definitions in maven configuration when building libhdfspp
[ https://issues.apache.org/jira/browse/HDFS-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014961#comment-15014961 ] Allen Wittenauer edited comment on HDFS-9416 at 11/20/15 9:23 PM: -- Today: http://github.com/apache/yetus/blob/master/precommit/test-patch-docker/Dockerfile Future: http://github.com/apache/hadoop/blob/trunk/dev-support/docker/Dockerfile Both are wherever the OS installs it. was (Author: aw): Today: https://github.com/apache/yetus/blob/master/precommit/test-patch-docker/Dockerfile Future: https://github.com/apache/hadoop/blob/trunk/dev-support/docker/Dockerfile Both are wherever the OS installs it. > Respect OpenSSL and protobuf definitions in maven configuration when building > libhdfspp > --- > > Key: HDFS-9416 > URL: https://issues.apache.org/jira/browse/HDFS-9416 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Haohui Mai >Assignee: Xiaobing Zhou >Priority: Blocker > Attachments: HDFS-9416.004.patch, HDFS-9416.HDFS-8707.004.patch, > HDFS-9416.HDFS-8707.005.patch > > > As discovered in HDFS-9380 the current pom.xml / CMakeLists.txt in libhdfspp > does not respect the configuration from the maven command line. Subsequently > it breaks the Jenkins build. > Both pom.xml and CMakeLists.txt need to be fixed to get Jenkins working again. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode
[
https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018828#comment-15018828
]
Nathan Roberts commented on HDFS-7060:
--
Has anyone given any further thought on this patch? It seems safe to me and it
eliminates serious stability issues when datanodes' disks get very busy.
_getUsed()_ returns a value that is calculated by a DU process that ran for a
long time anyway, (so it's always somewhat out-of-sync). It's not very
difficult to load up a datanode with disk-intensive tasks that prevent the
datanode from getting a heartbeat in for several minutes, eventually being
declared dead by the NN. We've seen this take out entire clusters with large
Map/Reduce merges, as well as very large shuffles.
> Avoid taking locks when sending heartbeats from the DataNode
>
>
> Key: HDFS-7060
> URL: https://issues.apache.org/jira/browse/HDFS-7060
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Xinwei Qin
> Labels: BB2015-05-TBR
> Attachments: HDFS-7060-002.patch, HDFS-7060.000.patch,
> HDFS-7060.001.patch
>
>
> We're seeing the heartbeat is blocked by the monitor of {{FsDatasetImpl}}
> when the DN is under heavy load of writes:
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getDfsUsed(FsVolumeImpl.java:115)
> - waiting to lock <0x000780304fb8> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getStorageReports(FsDatasetImpl.java:91)
> - locked <0x000780612fd8> (a java.lang.Object)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:563)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:668)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:827)
> at java.lang.Thread.run(Thread.java:744)
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:743)
> - waiting to lock <0x000780304fb8> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:60)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:169)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:621)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:232)
> at java.lang.Thread.run(Thread.java:744)
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1006)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:59)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:244)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:195)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:753)
> - locked <0x000780304fb8> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:60)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:169)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:621)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:232)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-9038) Reserved space is erroneously counted towards non-DFS used.
[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018761#comment-15018761
]
Arpit Agarwal edited comment on HDFS-9038 at 11/20/15 8:55 PM:
---
Hi [~brahmareddy], thanks for taking this up. The approach looks good.
# The calculation in FsVolumeImpl#getNonDfsUsed looks wrong. Won't it always be
zero? The correct calculation should replace getAvailable() with actual volume
free space. Something like
{{Files.getFileStore(currentDir.toPath()).getUnallocatedSpace()}} perhaps. Also
I think reservedForReplicas need not be subtracted in the nonDFSUsed
calculation, since it is purely a bytes on disk calculation.
{code}
public long getAvailable() throws IOException {
long remaining = getCapacity() - getDfsUsed() - reservedForReplicas.get();
...
public long getNonDfsUsed() throws IOException {
long nonDFSUsed = getCapacity() - getDfsUsed() - getAvailable()
- reservedForReplicas.get();
{code}
# Can you add a targeted unit test for the above calculation? A mock-based test
would be ideal.
# If the DN does not report nonDfsUsed, the NN should initialize it using the
older calculation. Else the NN will report nonDfsUsed as zero for older DNs
e.g. during upgrade. The fix can be done in
{{PBHelperClient#convert(DatanodeInfoProto)}}.
# We can expose nonDfsUsed via StorageTypeStats and DatanodeStatistics. Okay to
do in a separate Jira.
# Also we can expose reservedForReplicas in a separate Jira. Previously this
space was accounted for in nonDfsUsed but now it will not be accounted for
anywhere.
# Unrelated - we should replace {{DatanodeInfo}} constructors with a builder
pattern in a separate Jira.
was (Author: arpitagarwal):
Hi [~brahmareddy], thanks for taking this up. The approach looks good.
# The calculation in FsVolumeImpl#getNonDfsUsed looks wrong. Won't it always be
zero? The correct calculation should replace getAvailable() with actual volume
free space. Something like
{{Files.getFileStore(currentDir.toPath()).getUnallocatedSpace()}} perhaps.
{code}
public long getAvailable() throws IOException {
long remaining = getCapacity() - getDfsUsed() - reservedForReplicas.get();
...
public long getNonDfsUsed() throws IOException {
long nonDFSUsed = getCapacity() - getDfsUsed() - getAvailable()
- reservedForReplicas.get();
{code}
# Can you add a targeted unit test for the above calculation? A mock-based test
would be ideal.
# If the DN does not report nonDfsUsed, the NN should initialize it using the
older calculation. Else the NN will report nonDfsUsed as zero for older DNs
e.g. during upgrade. The fix can be done in
{{PBHelperClient#convert(DatanodeInfoProto)}}.
# We can expose nonDfsUsed via StorageTypeStats and DatanodeStatistics. Okay to
do in a separate Jira.
# Also we can expose reservedForReplicas in a separate Jira. Previously this
space was accounted for in nonDfsUsed but now it will not be accounted for
anywhere.
# Unrelated - we should replace {{DatanodeInfo}} constructors with a builder
pattern in a separate Jira.
> Reserved space is erroneously counted towards non-DFS used.
> ---
>
> Key: HDFS-9038
> URL: https://issues.apache.org/jira/browse/HDFS-9038
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Chris Nauroth
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch,
> HDFS-9038-004.patch, HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider
> the reserved space held by the {{dfs.datanode.du.reserved}} configuration
> property. As a side effect, reserved space is now counted towards non-DFS
> used. I don't believe it was intentional to change the definition of non-DFS
> used. This issue proposes restoring the prior behavior: do not count
> reserved space towards non-DFS used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9038) Reserved space is erroneously counted towards non-DFS used.
[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018761#comment-15018761
]
Arpit Agarwal commented on HDFS-9038:
-
Hi [~brahmareddy], thanks for taking this up. The approach looks good.
# The calculation in FsVolumeImpl#getNonDfsUsed looks wrong. Won't it always be
zero? The correct calculation should replace getAvailable() with actual volume
free space. Something like
{{Files.getFileStore(currentDir.toPath()).getUnallocatedSpace()}} perhaps.
{code}
public long getAvailable() throws IOException {
long remaining = getCapacity() - getDfsUsed() - reservedForReplicas.get();
...
public long getNonDfsUsed() throws IOException {
long nonDFSUsed = getCapacity() - getDfsUsed() - getAvailable()
- reservedForReplicas.get();
{code}
# Can you add a targeted unit test for the above calculation? A mock-based test
would be ideal.
# If the DN does not report nonDfsUsed, the NN should initialize it using the
older calculation. Else the NN will report nonDfsUsed as zero for older DNs
e.g. during upgrade. The fix can be done in
{{PBHelperClient#convert(DatanodeInfoProto)}}.
# We can expose nonDfsUsed via StorageTypeStats and DatanodeStatistics. Okay to
do in a separate Jira.
# Also we can expose reservedForReplicas in a separate Jira. Previously this
space was accounted for in nonDfsUsed but now it will not be accounted for
anywhere.
# Unrelated - we should replace {{DatanodeInfo}} constructors with a builder
pattern in a separate Jira.
> Reserved space is erroneously counted towards non-DFS used.
> ---
>
> Key: HDFS-9038
> URL: https://issues.apache.org/jira/browse/HDFS-9038
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Chris Nauroth
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch,
> HDFS-9038-004.patch, HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider
> the reserved space held by the {{dfs.datanode.du.reserved}} configuration
> property. As a side effect, reserved space is now counted towards non-DFS
> used. I don't believe it was intentional to change the definition of non-DFS
> used. This issue proposes restoring the prior behavior: do not count
> reserved space towards non-DFS used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9359) Test libhdfs++ with existing libhdfs tests
[
https://issues.apache.org/jira/browse/HDFS-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018756#comment-15018756
]
Hadoop QA commented on HDFS-9359:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
10s {color} | {color:green} HDFS-8707 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 17s
{color} | {color:red} hadoop-hdfs-native-client in HDFS-8707 failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-hdfs-native-client in HDFS-8707 failed with JDK
v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s
{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 11s
{color} | {color:red} hadoop-hdfs-native-client in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 11s {color} |
{color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 11s {color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-hdfs-native-client in the patch failed with JDK
v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 13s {color} |
{color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.7.0_85.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 13s {color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.7.0_85.
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 12s {color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 13s {color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.7.0_85.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 26s
{color} | {color:red} Patch generated 427 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 24s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:date2015-11-20 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12773583/HDFS-9359.HDFS-8707.001.patch
|
| JIRA Issue | HDFS-9359 |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux a9ae20f1bd3f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh
|
| git revision | HDFS-8707 / 68f4366 |
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/13583/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_66.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/13583/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop
[jira] [Commented] (HDFS-9441) Do not call construct path string when choosing block placement targets
[ https://issues.apache.org/jira/browse/HDFS-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018732#comment-15018732 ] Tsz Wo Nicholas Sze commented on HDFS-9441: --- There are other callers passing src as a String such as chooseTarget4WebHDFS and chooseTarget4AdditionalDatanode. We may not need to resolve path to inode in those cases. > Do not call construct path string when choosing block placement targets > --- > > Key: HDFS-9441 > URL: https://issues.apache.org/jira/browse/HDFS-9441 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Attachments: h9441_20151118.patch, h9441_20151119.patch > > > - INodeFile.getName() is expensive since it involves quite a few string > operations. The method is called in both ReplicationWork and > ErasureCodingWork but the default BlockPlacementPolicy does not use the > returned string. We should simply pass BlockCollection to reduce unnecessary > computation when using the default BlockPlacementPolicy. > - Another improvement: the return type of FSNamesystem.getBlockCollection > should be changed to INodeFile since it always returns an INodeFile object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6101) TestReplaceDatanodeOnFailure fails occasionally
[ https://issues.apache.org/jira/browse/HDFS-6101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-6101: -- Attachment: HDFS-6101.005.patch I am attach rev5, which reduced the number of writers to 5. > TestReplaceDatanodeOnFailure fails occasionally > --- > > Key: HDFS-6101 > URL: https://issues.apache.org/jira/browse/HDFS-6101 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Arpit Agarwal >Assignee: Wei-Chiu Chuang > Attachments: HDFS-6101.001.patch, HDFS-6101.002.patch, > HDFS-6101.003.patch, HDFS-6101.004.patch, HDFS-6101.005.patch, > TestReplaceDatanodeOnFailure.log > > > Exception details in a comment below. > The failure repros on both OS X and Linux if I run the test ~10 times in a > loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9314) Improve BlockPlacementPolicyDefault's picking of excess replicas
[
https://issues.apache.org/jira/browse/HDFS-9314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-9314:
Attachment: HDFS-9314.004.patch
Thanks again [~mingma]! Attached patch 4 with the aforementioned changes:
- Pass in {{rackMap}} to {{pickupReplicaSet}}, add param to methods along the
way.
- Update comments in {{BlockPlacementPolicyWithUpgradeDomain}}.
- Add test case to {{TestReplicationPolicyWithUpgradeDomain}}, verified it
failed before the fix and passed after.
> Improve BlockPlacementPolicyDefault's picking of excess replicas
>
>
> Key: HDFS-9314
> URL: https://issues.apache.org/jira/browse/HDFS-9314
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Xiao Chen
> Attachments: HDFS-9314.001.patch, HDFS-9314.002.patch,
> HDFS-9314.003.patch, HDFS-9314.004.patch
>
>
> The test case used in HDFS-9313 identified NullPointerException as well as
> the limitation of excess replica picking. If the current replicas are on
> {SSD(rack r1), DISK(rack 2), DISK(rack 3), DISK(rack 3)} and the storage
> policy changes to HOT_STORAGE_POLICY_ID, BlockPlacementPolicyDefault's won't
> be able to delete SSD replica.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6101) TestReplaceDatanodeOnFailure fails occasionally
[ https://issues.apache.org/jira/browse/HDFS-6101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018639#comment-15018639 ] Wei-Chiu Chuang commented on HDFS-6101: --- Thanks for the comments and reviews. I was hesitate to continue on it because on my local machine, this test still frequently failed after I made the change suggested by [~walter.k.su]. The only way I could avoid failures is to reduce the number of concurrent writers. Originally the test had 10 writers, and it fails 1 in 4 times. I reduced the number to 5, and it did not fail for 100 runs. This is why I suspect there is another issue with the default block placement policy (HDFS-9361 Default block placement policy causes TestReplaceDataNodeOnFailure to fail intermittently) When there are 10 writers begin to writer at the same time, the policy will not allow some writers set up pipelines with 3 data nodes, due to the load factor of data nodes. When the writers reduces, the load reduces and therefore the test passed. > TestReplaceDatanodeOnFailure fails occasionally > --- > > Key: HDFS-6101 > URL: https://issues.apache.org/jira/browse/HDFS-6101 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Arpit Agarwal >Assignee: Wei-Chiu Chuang > Attachments: HDFS-6101.001.patch, HDFS-6101.002.patch, > HDFS-6101.003.patch, HDFS-6101.004.patch, HDFS-6101.005.patch, > TestReplaceDatanodeOnFailure.log > > > Exception details in a comment below. > The failure repros on both OS X and Linux if I run the test ~10 times in a > loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9359) Test libhdfs++ with existing libhdfs tests
[ https://issues.apache.org/jira/browse/HDFS-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated HDFS-9359: -- Status: Patch Available (was: Open) > Test libhdfs++ with existing libhdfs tests > -- > > Key: HDFS-9359 > URL: https://issues.apache.org/jira/browse/HDFS-9359 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Stephen > Attachments: HDFS-9359.HDFS-8707.001.patch > > > Leverage existing libhdfs tests, such as test_libhdfs_threaded.c, to exercise > functionality implemented in libhdfs\+\+. The remainder of functionality > needed to run the tests can be supported by delegating to libhdfs. This > allows functions implemented in libhdfs++ to be tested prior to implementing > the entirety of the hdfs c-api (i.e. reads can be tested prior to > implementing writes). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9359) Test libhdfs++ with existing libhdfs tests
[ https://issues.apache.org/jira/browse/HDFS-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated HDFS-9359: -- Attachment: HDFS-9359.HDFS-8707.001.patch hdfs_shim.c is created to support testing part of libhdfs++ that have been implemented. Functions not implemented in libhdfs++ are delegated to libhdfs. HDFS-9359.HDFS-8707.001.patch is mostly mechanically generated #defines (and corresponding #undefs) to rename symbols. Other ways of doing this were platform or compiler specific (e.g. objcopy or -fprefix-function-name). > Test libhdfs++ with existing libhdfs tests > -- > > Key: HDFS-9359 > URL: https://issues.apache.org/jira/browse/HDFS-9359 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Stephen > Attachments: HDFS-9359.HDFS-8707.001.patch > > > Leverage existing libhdfs tests, such as test_libhdfs_threaded.c, to exercise > functionality implemented in libhdfs\+\+. The remainder of functionality > needed to run the tests can be supported by delegating to libhdfs. This > allows functions implemented in libhdfs++ to be tested prior to implementing > the entirety of the hdfs c-api (i.e. reads can be tested prior to > implementing writes). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9144) Refactor libhdfs into stateful/ephemeral objects
[ https://issues.apache.org/jira/browse/HDFS-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018585#comment-15018585 ] James Clampffer commented on HDFS-9144: --- The rebased changes look good to me. Hopefully the changes due to DN retry weren't too painful. Looking forward to seeing the next patch. > Refactor libhdfs into stateful/ephemeral objects > > > Key: HDFS-9144 > URL: https://issues.apache.org/jira/browse/HDFS-9144 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS-8707 >Reporter: Bob Hansen >Assignee: Bob Hansen > Attachments: HDFS-9144.HDFS-8707.001.patch, > HDFS-9144.HDFS-8707.002.patch, HDFS-9144.HDFS-8707.003.patch, > HDFS-9144.HDFS-8707.004.patch > > > In discussion for other efforts, we decided that we should separate several > concerns: > * A posix-like FileSystem/FileHandle object (stream-based, positional reads) > * An ephemeral ReadOperation object that holds the state for > reads-in-progress, which consumes > * An immutable FileInfo object which holds the block map and file size (and > other metadata about the file that we assume will not change over the life of > the file) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9336) deleteSnapshot throws NPE when snapshotname is null
[
https://issues.apache.org/jira/browse/HDFS-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018565#comment-15018565
]
Brahma Reddy Battula commented on HDFS-9336:
[~ajisakaa] thanks a lot for taking a look into this issue.uploaded the patch
to separate the testcase..kindly review..
> deleteSnapshot throws NPE when snapshotname is null
> ---
>
> Key: HDFS-9336
> URL: https://issues.apache.org/jira/browse/HDFS-9336
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9336-002.patch, HDFS-9336-003.patch, HDFS-9336.patch
>
>
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$DeleteSnapshotRequestProto$Builder.setSnapshotName(ClientNamenodeProtocolProtos.java:17509)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.deleteSnapshot(ClientNamenodeProtocolTranslatorPB.java:1005)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:255)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy15.deleteSnapshot(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.deleteSnapshot(DFSClient.java:2106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1660)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.deleteSnapshot(DistributedFileSystem.java:1677)
> at
> org.apache.hadoop.hdfs.web.TestWebHDFS.testWebHdfsAllowandDisallowSnapshots(TestWebHDFS.java:380)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9336) deleteSnapshot throws NPE when snapshotname is null
[
https://issues.apache.org/jira/browse/HDFS-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-9336:
---
Attachment: HDFS-9336-003.patch
> deleteSnapshot throws NPE when snapshotname is null
> ---
>
> Key: HDFS-9336
> URL: https://issues.apache.org/jira/browse/HDFS-9336
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9336-002.patch, HDFS-9336-003.patch, HDFS-9336.patch
>
>
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$DeleteSnapshotRequestProto$Builder.setSnapshotName(ClientNamenodeProtocolProtos.java:17509)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.deleteSnapshot(ClientNamenodeProtocolTranslatorPB.java:1005)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:255)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy15.deleteSnapshot(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.deleteSnapshot(DFSClient.java:2106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1660)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.deleteSnapshot(DistributedFileSystem.java:1677)
> at
> org.apache.hadoop.hdfs.web.TestWebHDFS.testWebHdfsAllowandDisallowSnapshots(TestWebHDFS.java:380)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9038) Reserved space is erroneously counted towards non-DFS used.
[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018549#comment-15018549
]
Brahma Reddy Battula commented on HDFS-9038:
This is not failing locally.But there is chance that this testcase can fail
intermittently after this change(There is a chance that nonDFS usage might have
slightly due to testlogs,).So I am keeping safeside as I did
{{TestNamenodeCapacityReport.java}}
*Check following results for same*
||Testenv||dfsused||dfsremaining||nondfsused||TotalCapacity||difference||
|Eclipse| 888|570023337984|74221730952|644245069824|0|
|jenkins Run| 147456|20430970142720|2887306715136|23318276997120|8192|
> Reserved space is erroneously counted towards non-DFS used.
> ---
>
> Key: HDFS-9038
> URL: https://issues.apache.org/jira/browse/HDFS-9038
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Chris Nauroth
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch,
> HDFS-9038-004.patch, HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider
> the reserved space held by the {{dfs.datanode.du.reserved}} configuration
> property. As a side effect, reserved space is now counted towards non-DFS
> used. I don't believe it was intentional to change the definition of non-DFS
> used. This issue proposes restoring the prior behavior: do not count
> reserved space towards non-DFS used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9038) Reserved space is erroneously counted towards non-DFS used.
[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-9038:
---
Attachment: HDFS-9038-004.patch
> Reserved space is erroneously counted towards non-DFS used.
> ---
>
> Key: HDFS-9038
> URL: https://issues.apache.org/jira/browse/HDFS-9038
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Chris Nauroth
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch,
> HDFS-9038-004.patch, HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider
> the reserved space held by the {{dfs.datanode.du.reserved}} configuration
> property. As a side effect, reserved space is now counted towards non-DFS
> used. I don't believe it was intentional to change the definition of non-DFS
> used. This issue proposes restoring the prior behavior: do not count
> reserved space towards non-DFS used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8855) Webhdfs client leaks active NameNode connections
[ https://issues.apache.org/jira/browse/HDFS-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-8855: Attachment: HDFS-8855.009.patch Posted a new patch v009 to remove white spaces. The UT failures are not related. > Webhdfs client leaks active NameNode connections > > > Key: HDFS-8855 > URL: https://issues.apache.org/jira/browse/HDFS-8855 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Bob Hansen >Assignee: Xiaobing Zhou > Fix For: 2.8.0 > > Attachments: HDFS-8855.005.patch, HDFS-8855.006.patch, > HDFS-8855.007.patch, HDFS-8855.008.patch, HDFS-8855.009.patch, > HDFS-8855.1.patch, HDFS-8855.2.patch, HDFS-8855.3.patch, HDFS-8855.4.patch, > HDFS_8855.prototype.patch > > > The attached script simulates a process opening ~50 files via webhdfs and > performing random reads. Note that there are at most 50 concurrent reads, > and all webhdfs sessions are kept open. Each read is ~64k at a random > position. > The script periodically (once per second) shells into the NameNode and > produces a summary of the socket states. For my test cluster with 5 nodes, > it took ~30 seconds for the NameNode to have ~25000 active connections and > fails. > It appears that each request to the webhdfs client is opening a new > connection to the NameNode and keeping it open after the request is complete. > If the process continues to run, eventually (~30-60 seconds), all of the > open connections are closed and the NameNode recovers. > This smells like SoftReference reaping. Are we using SoftReferences in the > webhdfs client to cache NameNode connections but never re-using them? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9416) Respect OpenSSL and protobuf definitions in maven configuration when building libhdfspp
[ https://issues.apache.org/jira/browse/HDFS-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-9416: Attachment: HDFS-9416.HDFS-8707.005.patch Note that the patch v004 works well in my test environment, though it's not exactly identical to the build machine. I posted this patch v005 to follow hadoop-common pattern to locate openssl with a hope to get it work. > Respect OpenSSL and protobuf definitions in maven configuration when building > libhdfspp > --- > > Key: HDFS-9416 > URL: https://issues.apache.org/jira/browse/HDFS-9416 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Haohui Mai >Assignee: Xiaobing Zhou >Priority: Blocker > Attachments: HDFS-9416.004.patch, HDFS-9416.HDFS-8707.004.patch, > HDFS-9416.HDFS-8707.005.patch > > > As discovered in HDFS-9380 the current pom.xml / CMakeLists.txt in libhdfspp > does not respect the configuration from the maven command line. Subsequently > it breaks the Jenkins build. > Both pom.xml and CMakeLists.txt need to be fixed to get Jenkins working again. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9441) Do not call construct path string when choosing block placement targets
[
https://issues.apache.org/jira/browse/HDFS-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018477#comment-15018477
]
Kihwal Lee commented on HDFS-9441:
--
Any particular reason for making it {{Object}} instead of {{BlockCollection}}?
> Do not call construct path string when choosing block placement targets
> ---
>
> Key: HDFS-9441
> URL: https://issues.apache.org/jira/browse/HDFS-9441
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Attachments: h9441_20151118.patch, h9441_20151119.patch
>
>
> - INodeFile.getName() is expensive since it involves quite a few string
> operations. The method is called in both ReplicationWork and
> ErasureCodingWork but the default BlockPlacementPolicy does not use the
> returned string. We should simply pass BlockCollection to reduce unnecessary
> computation when using the default BlockPlacementPolicy.
> - Another improvement: the return type of FSNamesystem.getBlockCollection
> should be changed to INodeFile since it always returns an INodeFile object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9426) Rollingupgrade finalization is not backward compatible
[
https://issues.apache.org/jira/browse/HDFS-9426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-9426:
-
Status: Patch Available (was: Open)
> Rollingupgrade finalization is not backward compatible
> --
>
> Key: HDFS-9426
> URL: https://issues.apache.org/jira/browse/HDFS-9426
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Priority: Blocker
> Attachments: HDFS-9426.branch-2.7.poc.patch, HDFS-9426.trunk.poc.patch
>
>
> After HDFS-7645, the namenode can return non-null {{rollingUpgradeInfo}} in
> heatbeat reponses. 2.7.1 or 2.6.x datanodes won't finalize the upgrade
> because it's not null.
> NN might have to check the DN version and return different
> {{rollingUpgradeInfo}}.
> HDFS-8656 recognized the compatibility issue of the changed semantics, but
> unfortunately did not address the semantics of the heartbeat response.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9144) Refactor libhdfs into stateful/ephemeral objects
[ https://issues.apache.org/jira/browse/HDFS-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018351#comment-15018351 ] Bob Hansen commented on HDFS-9144: -- [~James Clampffer]: thanks for the review I have attached another version that is rebased on the DN retry stuff. There were a lot of conflicts if you want to give it another once-over. I will address your other comments in a further patch; you can wait for them to re-review if you would like. Thanks for taking the time to slog through all the changes. > Refactor libhdfs into stateful/ephemeral objects > > > Key: HDFS-9144 > URL: https://issues.apache.org/jira/browse/HDFS-9144 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS-8707 >Reporter: Bob Hansen >Assignee: Bob Hansen > Attachments: HDFS-9144.HDFS-8707.001.patch, > HDFS-9144.HDFS-8707.002.patch, HDFS-9144.HDFS-8707.003.patch, > HDFS-9144.HDFS-8707.004.patch > > > In discussion for other efforts, we decided that we should separate several > concerns: > * A posix-like FileSystem/FileHandle object (stream-based, positional reads) > * An ephemeral ReadOperation object that holds the state for > reads-in-progress, which consumes > * An immutable FileInfo object which holds the block map and file size (and > other metadata about the file that we assume will not change over the life of > the file) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9144) Refactor libhdfs into stateful/ephemeral objects
[ https://issues.apache.org/jira/browse/HDFS-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen updated HDFS-9144: - Attachment: HDFS-9144.HDFS-8707.004.patch > Refactor libhdfs into stateful/ephemeral objects > > > Key: HDFS-9144 > URL: https://issues.apache.org/jira/browse/HDFS-9144 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS-8707 >Reporter: Bob Hansen >Assignee: Bob Hansen > Attachments: HDFS-9144.HDFS-8707.001.patch, > HDFS-9144.HDFS-8707.002.patch, HDFS-9144.HDFS-8707.003.patch, > HDFS-9144.HDFS-8707.004.patch > > > In discussion for other efforts, we decided that we should separate several > concerns: > * A posix-like FileSystem/FileHandle object (stream-based, positional reads) > * An ephemeral ReadOperation object that holds the state for > reads-in-progress, which consumes > * An immutable FileInfo object which holds the block map and file size (and > other metadata about the file that we assume will not change over the life of > the file) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9445) Deadlock in datanode
[
https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018295#comment-15018295
]
Kihwal Lee commented on HDFS-9445:
--
And the stack trace:
{noformat}
Java stack information for the threads listed above:
===
"DataXceiver for client DFSClient_attempt_xxx [Sending block
BP-x:blk_123_456]":
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:234)
- waiting to lock <0xd60d9930> (a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:537)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
at java.lang.Thread.run(Thread.java:745)
"Thread-565":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xd55613c8> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.readLock(BPOfferService.java:105)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:166)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.checkBlock(BPOfferService.java:249)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.notifyNamenodeDeletedBlock(BPOfferService.java:255)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.notifyNamenodeDeletedBlock(DataNode.java:976)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:1891)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.removeVolumes(FsDatasetImpl.java:485)
- locked <0xd60d9930> (a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.removeVolumes(DataNode.java:690)
- locked <0xd58b9e70> (a
org.apache.hadoop.hdfs.server.datanode.DataNode)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.checkDiskError(DataNode.java:3137)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.access$800(DataNode.java:242)
at
org.apache.hadoop.hdfs.server.datanode.DataNode$7.run(DataNode.java:3166)
at java.lang.Thread.run(Thread.java:745)
"DataNode: heartbeating to my-nn:8020":
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.validateBlockFile(FsDatasetImpl.java:1741)
- waiting to lock <0xd60d9930> (a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockFile(FsDatasetImpl.java:663)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockFile(FsDatasetImpl.java:656)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getLength(FsDatasetImpl.java:649)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.checkBlock(FsDatasetImpl.java:1701)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.transferBlock(DataNode.java:1875)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.transferBlocks(DataNode.java:1931)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:657)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:615)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:858)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:672)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:824)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> Deadlock in datanode
>
>
> Key: HDFS-9445
> URL: https://issues.apache.org/jira/browse/HDFS-9445
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.2
>Re
[jira] [Updated] (HDFS-9445) Deadlock in datanode
[
https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-9445:
-
Description:
{noformat}
Found one Java-level deadlock:
=
"DataXceiver for client DFSClient_attempt_xxx at /1.2.3.4:100 [Sending block
BP-x:blk_123_456]":
waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
which is held by "Thread-565"
"Thread-565":
waiting for ownable synchronizer 0xd55613c8, (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "DataNode: heartbeating to my-nn:8020"
"DataNode: heartbeating to my-nn:8020":
waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
which is held by "Thread-565"
{noformat}
was:
{noformat}
Found one Java-level deadlock:
=
"DataXceiver for client DFSClient_attempt_xxx at /10.217.77.219:52900 [Sending
block BP-x:blk_123_456]":
waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
which is held by "Thread-565"
"Thread-565":
waiting for ownable synchronizer 0xd55613c8, (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "DataNode: heartbeating to my-nn:8020"
"DataNode: heartbeating to my-nn:8020":
waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
which is held by "Thread-565"
{noformat}
> Deadlock in datanode
>
>
> Key: HDFS-9445
> URL: https://issues.apache.org/jira/browse/HDFS-9445
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Kihwal Lee
>Priority: Blocker
>
> {noformat}
> Found one Java-level deadlock:
> =
> "DataXceiver for client DFSClient_attempt_xxx at /1.2.3.4:100 [Sending block
> BP-x:blk_123_456]":
> waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
> which is held by "Thread-565"
> "Thread-565":
> waiting for ownable synchronizer 0xd55613c8, (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
> which is held by "DataNode: heartbeating to my-nn:8020"
> "DataNode: heartbeating to my-nn:8020":
> waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
> which is held by "Thread-565"
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9445) Deadlock in datanode
[
https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-9445:
-
Affects Version/s: 2.7.2
> Deadlock in datanode
>
>
> Key: HDFS-9445
> URL: https://issues.apache.org/jira/browse/HDFS-9445
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Kihwal Lee
>Priority: Blocker
>
> {noformat}
> Found one Java-level deadlock:
> =
> "DataXceiver for client DFSClient_attempt_xxx at /10.217.77.219:52900
> [Sending block BP-x:blk_123_456]":
> waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
> which is held by "Thread-565"
> "Thread-565":
> waiting for ownable synchronizer 0xd55613c8, (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
> which is held by "DataNode: heartbeating to my-nn:8020"
> "DataNode: heartbeating to my-nn:8020":
> waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
> which is held by "Thread-565"
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9445) Deadlock in datanode
Kihwal Lee created HDFS-9445:
Summary: Deadlock in datanode
Key: HDFS-9445
URL: https://issues.apache.org/jira/browse/HDFS-9445
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Priority: Blocker
{noformat}
Found one Java-level deadlock:
=
"DataXceiver for client DFSClient_attempt_xxx at /10.217.77.219:52900 [Sending
block BP-x:blk_123_456]":
waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
which is held by "Thread-565"
"Thread-565":
waiting for ownable synchronizer 0xd55613c8, (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "DataNode: heartbeating to my-nn:8020"
"DataNode: heartbeating to my-nn:8020":
waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
which is held by "Thread-565"
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9407) TestFileTruncate fails with BindException
[
https://issues.apache.org/jira/browse/HDFS-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018224#comment-15018224
]
Brahma Reddy Battula commented on HDFS-9407:
bq. Yes it is unrelated, but it is another BindException. Some previous test
did not shut down NameNode properly. May be useful to investigate this in
another jira.
[~shv] HDFS-9444 raised to investigate on this.
> TestFileTruncate fails with BindException
> -
>
> Key: HDFS-9407
> URL: https://issues.apache.org/jira/browse/HDFS-9407
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9407-002.patch, HDFS-9407.patch
>
>
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/2530/
> {noformat}
> java.net.BindException: Problem binding to [localhost:8020]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
> at
> org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.setUp(TestFileTruncate.java:103)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9444) TestEditLogTailer#testNN1TriggersLogRolls fails bind exception.
Brahma Reddy Battula created HDFS-9444:
--
Summary: TestEditLogTailer#testNN1TriggersLogRolls fails bind
exception.
Key: HDFS-9444
URL: https://issues.apache.org/jira/browse/HDFS-9444
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Brahma Reddy Battula
https://builds.apache.org/job/Hadoop-Hdfs-trunk/2556/testReport/
{noformat}
java.net.BindException: Problem binding to [localhost:42477]
java.net.BindException: Address already in use; For more details see:
http://wiki.apache.org/hadoop/BindException
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.hadoop.ipc.Server.bind(Server.java:469)
at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
at org.apache.hadoop.ipc.Server.(Server.java:2464)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
at
org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
at
org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRolls(TestEditLogTailer.java:139)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testNN1TriggersLogRolls(TestEditLogTailer.java:114)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6937) Another issue in handling checksum errors in write pipeline
[
https://issues.apache.org/jira/browse/HDFS-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018177#comment-15018177
]
Mikhail Dutikov commented on HDFS-6937:
---
Good day, any update on this? I seem to be running into a similar issue with
Hbase WAL based on HDFS 2.6 chd 2.4.8. A pipeline is being reconstructed with
many candidate datanodes, but none of the substitutes seems to be receiving the
block replica correctly:
Checksum error in block from /IP:port
org.apache.hadoop.fs.ChecksumException: Checksum error:
DFSClient_NONMAPREDUCE_1267344484_1 at 2048 exp: -652491368 got: -585724081
[note: checksums are the same on all new candidate nodes]
java.io.IOException: Terminating due to a checksum error.java.io.IOException:
Unexpected checksum mismatch while writing from /IP:port
>···at
>org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:562)
>···at
>org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:780)
>···at
>org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:783)
>···at
>org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
>···at
>org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
>···at
>org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:243)
>···at java.lang.Thread.run(Thread.java:745)
This is repeated until no datanode is left to try, and leads to data loss (?)
and terminated Region Servers due to inability to write to WAL. (The cluster
has 3 racks and 21 nodes)
> Another issue in handling checksum errors in write pipeline
> ---
>
> Key: HDFS-6937
> URL: https://issues.apache.org/jira/browse/HDFS-6937
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, hdfs-client
>Affects Versions: 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Colin Patrick McCabe
>
> Given a write pipeline:
> DN1 -> DN2 -> DN3
> DN3 detected cheksum error and terminate, DN2 truncates its replica to the
> ACKed size. Then a new pipeline is attempted as
> DN1 -> DN2 -> DN4
> DN4 detects checksum error again. Later when replaced DN4 with DN5 (and so
> on), it failed for the same reason. This led to the observation that DN2's
> data is corrupted.
> Found that the software currently truncates DN2's replca to the ACKed size
> after DN3 terminates. But it doesn't check the correctness of the data
> already written to disk.
> So intuitively, a solution would be, when downstream DN (DN3 here) found
> checksum error, propagate this info back to upstream DN (DN2 here), DN2
> checks the correctness of the data already written to disk, and truncate the
> replica to to MIN(correctDataSize, ACKedSize).
> Found this issue is similar to what was reported by HDFS-3875, and the
> truncation at DN2 was actually introduced as part of the HDFS-3875 solution.
> Filing this jira for the issue reported here. HDFS-3875 was filed by
> [~tlipcon]
> and found he proposed something similar there.
> {quote}
> if the tail node in the pipeline detects a checksum error, then it returns a
> special error code back up the pipeline indicating this (rather than just
> disconnecting)
> if a non-tail node receives this error code, then it immediately scans its
> own block on disk (from the beginning up through the last acked length). If
> it detects a corruption on its local copy, then it should assume that it is
> the faulty one, rather than the downstream neighbor. If it detects no
> corruption, then the faulty node is either the downstream mirror or the
> network link between the two, and the current behavior is reasonable.
> {quote}
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9038) Reserved space is erroneously counted towards non-DFS used.
[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15015584#comment-15015584
]
Vinayakumar B commented on HDFS-9038:
-
Please check the failure of
{{hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics}}
> Reserved space is erroneously counted towards non-DFS used.
> ---
>
> Key: HDFS-9038
> URL: https://issues.apache.org/jira/browse/HDFS-9038
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Chris Nauroth
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider
> the reserved space held by the {{dfs.datanode.du.reserved}} configuration
> property. As a side effect, reserved space is now counted towards non-DFS
> used. I don't believe it was intentional to change the definition of non-DFS
> used. This issue proposes restoring the prior behavior: do not count
> reserved space towards non-DFS used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-9426) Rollingupgrade finalization is not backward compatible
[
https://issues.apache.org/jira/browse/HDFS-9426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15015437#comment-15015437
]
Vinayakumar B edited comment on HDFS-9426 at 11/20/15 10:33 AM:
bq. I am just wondering whether it will be worth doing more code changes to
avoid introducing the redundant field in the new protocol. Vinayakumar B, what
do you think?
I am thinking current patch is sufficient for the fix. At-least to resolve this
blocker for 2.7.2.
Is there any other way, except this fix and reverting HDFS-7645?
was (Author: vinayrpet):
bq. I am just wondering whether it will be worth doing more code changes to
avoid introducing the redundant field in the new protocol. Vinayakumar B, what
do you think?
I am thinking current patch is sufficient for the fix. At-least to resolve this
blocker for 2.7.2.
Is there any other way, this fix and reverting HDFS-7645?
> Rollingupgrade finalization is not backward compatible
> --
>
> Key: HDFS-9426
> URL: https://issues.apache.org/jira/browse/HDFS-9426
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Priority: Blocker
> Attachments: HDFS-9426.branch-2.7.poc.patch, HDFS-9426.trunk.poc.patch
>
>
> After HDFS-7645, the namenode can return non-null {{rollingUpgradeInfo}} in
> heatbeat reponses. 2.7.1 or 2.6.x datanodes won't finalize the upgrade
> because it's not null.
> NN might have to check the DN version and return different
> {{rollingUpgradeInfo}}.
> HDFS-8656 recognized the compatibility issue of the changed semantics, but
> unfortunately did not address the semantics of the heartbeat response.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9038) Reserved space is erroneously counted towards non-DFS used.
[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15015526#comment-15015526
]
Hadoop QA commented on HDFS-9038:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 43s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 37s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 32s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 52s {color}
| {color:red} hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3
new issues (was 49, now 49). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 41s
{color} | {color:red} hadoop-hdfs-project-jdk1.7.0_79 with JDK v1.7.0_79
generated 3 new issues (was 51, now 51). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 25s
{color} | {color:red} Patch generated 4 new checkstyle issues in
hadoop-hdfs-project (total was 293, now 295). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 44s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 2s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 10s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 14s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1
[jira] [Updated] (HDFS-9428) Fix intermittent failure of TestDNFencing.testQueueingWithAppend
[ https://issues.apache.org/jira/browse/HDFS-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-9428: Status: Patch Available (was: Open) +1 pending jenkins. > Fix intermittent failure of TestDNFencing.testQueueingWithAppend > > > Key: HDFS-9428 > URL: https://issues.apache.org/jira/browse/HDFS-9428 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: HDFS-9428.001.patch, HDFS-9428.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9428) Fix intermittent failure of TestDNFencing.testQueueingWithAppend
[ https://issues.apache.org/jira/browse/HDFS-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-9428: Status: Open (was: Patch Available) > Fix intermittent failure of TestDNFencing.testQueueingWithAppend > > > Key: HDFS-9428 > URL: https://issues.apache.org/jira/browse/HDFS-9428 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: HDFS-9428.001.patch, HDFS-9428.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9426) Rollingupgrade finalization is not backward compatible
[
https://issues.apache.org/jira/browse/HDFS-9426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15015437#comment-15015437
]
Vinayakumar B commented on HDFS-9426:
-
bq. I am just wondering whether it will be worth doing more code changes to
avoid introducing the redundant field in the new protocol. Vinayakumar B, what
do you think?
I am thinking current patch is sufficient for the fix. At-least to resolve this
blocker for 2.7.2.
Is there any other way, this fix and reverting HDFS-7645?
> Rollingupgrade finalization is not backward compatible
> --
>
> Key: HDFS-9426
> URL: https://issues.apache.org/jira/browse/HDFS-9426
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Priority: Blocker
> Attachments: HDFS-9426.branch-2.7.poc.patch, HDFS-9426.trunk.poc.patch
>
>
> After HDFS-7645, the namenode can return non-null {{rollingUpgradeInfo}} in
> heatbeat reponses. 2.7.1 or 2.6.x datanodes won't finalize the upgrade
> because it's not null.
> NN might have to check the DN version and return different
> {{rollingUpgradeInfo}}.
> HDFS-8656 recognized the compatibility issue of the changed semantics, but
> unfortunately did not address the semantics of the heartbeat response.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9411) HDFS ZoneLabel support
[ https://issues.apache.org/jira/browse/HDFS-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15015401#comment-15015401 ] Mark Sun commented on HDFS-9411: Hi [~vinayrpet], sorry for late reply * *About use DataNode configuration or not.* ** I mixed up StorageType and DataNode StorageInfo, I means maybe DataNode can storage zone_label into VERSION file of each Volume, or another file in same directories. Sure, if then we have to add new command to heartbeat. ** Besides consideration we already discussed, set zone_label on Volume could be better in case of Volume level isolation in the future. ** zone_label updating is not like simple configuration applying, it is mostly looks like refreshNodes: *** Both NameNode and DataNode involved. *** In same case, we should ensure necessary works(block moving or erasuring) done before label finally changes. * *About BPP related API.* ** Your comments makes me a litter bit clear: *** Client do not set xattr explicitly, but ZoneProvider in NameNode will read policy from user-context and zone-stats, and then set zone layout as xattr to file. *** Use set/getZone API & CLI to change zone layout explicitly after file created, and new file/subdirs can inherit zone layout from its parent. ** Still not clear： *** Where does NameNode get ZoneProvider implementation of certain user? *** If NameNode stores zone layout to xattr, how to deal with zoneStats change? How to distinguish between “we want to store block in zone_a” and “we want to store block in zone_b, but zone_b is full according to zoneStats, so we use zone_a”? *** How to deal with large files? zoneStats always changes, will we update zone layout with new zoneStats for later blocks? ** Proposition *** We do not store xattr by NameNode, just re-calculate for each block allocation. *** If we want per-file policy, we should store **the policy itself** to xattr, not the layout result, due to zoneStats changes. *** Maybe we need to pass&store policies to DataNode also, for new zone based Balancer or Mover. * *About feature on-off control and DEFAULT_ZONE* ** Agree with you, it’s just a philosophy level tendency. I just think DEFAULT_ZONE can make API uniform with less if/else, and by default, every node belongs to DEFAULT_ZONE, and the code path will be same to current implementation. ** Or another option, we may use polymorphism to reduce if/else and code complexity caused by on-off control. ** Sure, both are OK :) * *Make zone transparent to existing BPPs* ** I proposed expand zone to node list to make zone transparent to BPPs, for: *** Code is more generic, the new constraint is BPP should put X replicas to set, but doesn’t restrict this constraint should be made by ZoneProvider. (We can even use this constraint to re-implementation local-node-choosing ) *** More flexible, in some ambident scenario, one replica can be placed to both zone_a and zone_b. *** Code is loosely-coupled, BPP code modification don’t required to know zone_label. > HDFS ZoneLabel support > -- > > Key: HDFS-9411 > URL: https://issues.apache.org/jira/browse/HDFS-9411 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS_ZoneLabels-16112015.pdf > > > HDFS currently stores data blocks on different datanodes chosen by > BlockPlacement Policy. These datanodes are random within the > scope(local-rack/different-rack/nodegroup) of network topology. > In Multi-tenant (Tenant can be user/service) scenario, blocks of any tenant > can be on any datanodes. > Based on applications of different tenant, sometimes datanode might get busy > making the other tenant's application to slow down. It would be better if > admin's have a provision to logically divide the cluster among multi-tenants. > ZONE_LABELS can logically divide the cluster datanodes into multiple Zones. > High level design doc to follow soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9427) HDFS should not default to ephemeral ports
[ https://issues.apache.org/jira/browse/HDFS-9427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15015395#comment-15015395 ] Aaron T. Myers commented on HDFS-9427: -- +1, let's make this happen. Seems long overdue to me. > HDFS should not default to ephemeral ports > -- > > Key: HDFS-9427 > URL: https://issues.apache.org/jira/browse/HDFS-9427 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Xiaobing Zhou > Labels: Incompatible > > HDFS defaults to ephemeral ports for the some HTTP/RPC endpoints. This can > cause bind exceptions on service startup if the port is in use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
