[jira] [Commented] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219404#comment-15219404
]
Hadoop QA commented on HDFS-9263:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HDFS-9263 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-9263 |
| GITHUB PR | https://github.com/apache/hadoop/pull/53 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15018/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch,
> HDFS-9263-003.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-9263:
Status: Patch Available (was: Open)
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch,
> HDFS-9263-003.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-9263:
Status: Open (was: Patch Available)
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch,
> HDFS-9263-003.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219403#comment-15219403
]
Hadoop QA commented on HDFS-9263:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s {color}
| {color:red} HDFS-9263 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-9263 |
| GITHUB PR | https://github.com/apache/hadoop/pull/53 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15017/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch,
> HDFS-9263-003.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-10239) Fsshell mv fails if port usage doesn't match in src and destination paths
[
https://issues.apache.org/jira/browse/HDFS-10239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuhu Shukla updated HDFS-10239:
---
Status: Patch Available (was: In Progress)
> Fsshell mv fails if port usage doesn't match in src and destination paths
> -
>
> Key: HDFS-10239
> URL: https://issues.apache.org/jira/browse/HDFS-10239
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-10239.001.patch
>
>
> If one of the src or destination fs URIs does not contain the port while the
> other one does, the MoveCommands#processPath preemptively throws
> PathIOException "Does not match target filesystem".
> eg.
> {code}
> -bash-4.1$ hadoop fs -mv hdfs://localhost:8020/tmp/foo3
> hdfs://localhost/tmp/foo4
> mv: `hdfs://localhost:8020:8020/tmp/foo3': Does not match target filesystem
> {code}
> This is due to strict string check in {{processPath}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HDFS-10239) Fsshell mv fails if port usage doesn't match in src and destination paths
[
https://issues.apache.org/jira/browse/HDFS-10239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-10239 started by Kuhu Shukla.
--
> Fsshell mv fails if port usage doesn't match in src and destination paths
> -
>
> Key: HDFS-10239
> URL: https://issues.apache.org/jira/browse/HDFS-10239
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-10239.001.patch
>
>
> If one of the src or destination fs URIs does not contain the port while the
> other one does, the MoveCommands#processPath preemptively throws
> PathIOException "Does not match target filesystem".
> eg.
> {code}
> -bash-4.1$ hadoop fs -mv hdfs://localhost:8020/tmp/foo3
> hdfs://localhost/tmp/foo4
> mv: `hdfs://localhost:8020:8020/tmp/foo3': Does not match target filesystem
> {code}
> This is due to strict string check in {{processPath}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-10239) Fsshell mv fails if port usage doesn't match in src and destination paths
[
https://issues.apache.org/jira/browse/HDFS-10239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuhu Shukla updated HDFS-10239:
---
Attachment: HDFS-10239.001.patch
Attaching initial patch.
> Fsshell mv fails if port usage doesn't match in src and destination paths
> -
>
> Key: HDFS-10239
> URL: https://issues.apache.org/jira/browse/HDFS-10239
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-10239.001.patch
>
>
> If one of the src or destination fs URIs does not contain the port while the
> other one does, the MoveCommands#processPath preemptively throws
> PathIOException "Does not match target filesystem".
> eg.
> {code}
> -bash-4.1$ hadoop fs -mv hdfs://localhost:8020/tmp/foo3
> hdfs://localhost/tmp/foo4
> mv: `hdfs://localhost:8020:8020/tmp/foo3': Does not match target filesystem
> {code}
> This is due to strict string check in {{processPath}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-10239) Fsshell mv fails if port usage doesn't match in src and destination paths
Kuhu Shukla created HDFS-10239:
--
Summary: Fsshell mv fails if port usage doesn't match in src and
destination paths
Key: HDFS-10239
URL: https://issues.apache.org/jira/browse/HDFS-10239
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.7.2
Reporter: Kuhu Shukla
Assignee: Kuhu Shukla
If one of the src or destination fs URIs does not contain the port while the
other one does, the MoveCommands#processPath preemptively throws
PathIOException "Does not match target filesystem".
eg.
{code}
-bash-4.1$ hadoop fs -mv hdfs://localhost:8020/tmp/foo3
hdfs://localhost/tmp/foo4
mv: `hdfs://localhost:8020:8020/tmp/foo3': Does not match target filesystem
{code}
This is due to strict string check in {{processPath}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219377#comment-15219377
]
Hadoop QA commented on HDFS-9263:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s {color}
| {color:red} HDFS-9263 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-9263 |
| GITHUB PR | https://github.com/apache/hadoop/pull/53 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15016/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch,
> HDFS-9263-003.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10238) Ozone : Add chunk persistance
[ https://issues.apache.org/jira/browse/HDFS-10238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219375#comment-15219375 ] Anu Engineer commented on HDFS-10238: - Test failures are not related to the patch. ASF warning is from trunk and there is already a patch. Checkstyle is an issue about missing package-info. > Ozone : Add chunk persistance > - > > Key: HDFS-10238 > URL: https://issues.apache.org/jira/browse/HDFS-10238 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-10238-HDFS-7240.001.patch > > > Add support for chunks in containers for ozone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-9263:
Attachment: HDFS-9263-003.patch
Attaching the Updated patch.
Replaced all occurances of "test.build.data" with GenericTestUtils
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch,
> HDFS-9263-003.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9214) Support reconfiguring dfs.datanode.balance.max.concurrent.moves without DN restart
[
https://issues.apache.org/jira/browse/HDFS-9214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219362#comment-15219362
]
Tsz Wo Nicholas Sze commented on HDFS-9214:
---
Please add release note to describe how to reconfigure
dfs.datanode.balance.max.concurrent.moves. Thanks.
> Support reconfiguring dfs.datanode.balance.max.concurrent.moves without DN
> restart
> --
>
> Key: HDFS-9214
> URL: https://issues.apache.org/jira/browse/HDFS-9214
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Fix For: 2.8.0
>
> Attachments: HDFS-9214.001.patch, HDFS-9214.002.patch,
> HDFS-9214.003.patch, HDFS-9214.004.patch, HDFS-9214.005.patch
>
>
> This is to reconfigure
> {code}
> dfs.datanode.balance.max.concurrent.moves
> {code} without restarting DN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9952) Expose FSNamesystem lock wait time as metrics
[
https://issues.apache.org/jira/browse/HDFS-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219354#comment-15219354
]
Hadoop QA commented on HDFS-9952:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 44s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 5 new +
228 unchanged - 0 fixed = 233 total (was 228) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 23s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 53m 40s
{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 136m 14s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests | hadoop.hdfs.TestHFlush |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795593/HDFS-9952-04.patch |
| JIRA Issue | HDFS-9952 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d6ab80aa574d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3
[jira] [Commented] (HDFS-9943) Support reconfiguring namenode replication confs
[
https://issues.apache.org/jira/browse/HDFS-9943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219341#comment-15219341
]
Hadoop QA commented on HDFS-9943:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 55s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 24s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 5 new +
290 unchanged - 4 fixed = 295 total (was 294) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 55s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 58s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 211m 52s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests |
hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.tools.TestDFSAdmin |
| JDK v1.7.0_95 Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| |
[jira] [Updated] (HDFS-10237) Support specifying checksum type in WebHDFS/HTTPFS writers
[ https://issues.apache.org/jira/browse/HDFS-10237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-10237: --- Attachment: HDFS-10237.001.patch * Fixed the javadoc extra param problem * Fixed logic in the ChecksumOptParam that was not considering the string value of "null" vs. a literal null, and was breaking the contract test/non-recursive-create The other tests appear unrelated. > Support specifying checksum type in WebHDFS/HTTPFS writers > -- > > Key: HDFS-10237 > URL: https://issues.apache.org/jira/browse/HDFS-10237 > Project: Hadoop HDFS > Issue Type: New Feature > Components: webhdfs >Affects Versions: 2.8.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Minor > Attachments: HDFS-10237.000.patch, HDFS-10237.001.patch > > > Currently you cannot set a desired checksum type over a WebHDFS or HTTPFS > writer, as you can with the regular DFS writer (done via HADOOP-8240) > This JIRA covers the changes necessary to bring the same ability to WebHDFS > and HTTPFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10238) Ozone : Add chunk persistance
[
https://issues.apache.org/jira/browse/HDFS-10238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219244#comment-15219244
]
Hadoop QA commented on HDFS-10238:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
36s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 15s
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 15s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 2s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 28s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 18s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 28s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 44s
{color} | {color:red} Patch generated 4 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 193m 41s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests |
hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.namenode.TestEditLog |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
[jira] [Commented] (HDFS-9918) Erasure Coding: Sort located striped blocks based on decommissioned states
[
https://issues.apache.org/jira/browse/HDFS-9918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219241#comment-15219241
]
Walter Su commented on HDFS-9918:
-
The optimization works for {{BlockInfoStriped}}. A missing block occupies a
slot. Like
{noformat}
0, null, 2, 3, 4, 5, 6, 7, 8, 1, 0', 1', 7', 8'
{noformat}
In LocatedStripedBlock, the data is like
{noformat}
0, 2, 3, 4, 5, 6, 7, 8, 1, 0', 1', 7', 8'
{noformat}
That's why I don't see the point maintain the order (of the in-service ones).
Unless we change {{createLocatedBlock(..)}}. But optimizing
{{LocatedStripedBlock}} to save some network traffic seems trival.
> Erasure Coding: Sort located striped blocks based on decommissioned states
> --
>
> Key: HDFS-9918
> URL: https://issues.apache.org/jira/browse/HDFS-9918
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-9918-001.patch, HDFS-9918-002.patch,
> HDFS-9918-003.patch, HDFS-9918-004.patch, HDFS-9918-005.patch,
> HDFS-9918-006.patch, HDFS-9918-007.patch, HDFS-9918-008.patch
>
>
> This jira is a follow-on work of HDFS-8786, where we do decommissioning of
> datanodes having striped blocks.
> Now, after decommissioning it requires to change the ordering of the storage
> list so that the decommissioned datanodes should only be last node in list.
> For example, assume we have a block group with storage list:-
> d0, d1, d2, d3, d4, d5, d6, d7, d8, d9
> mapping to indices
> 0, 1, 2, 3, 4, 5, 6, 7, 8, 2
> Here the internal block b2 is duplicated, locating in d2 and d9. If d2 is a
> decommissioning node then should switch d2 and d9 in the storage list.
> Thanks [~jingzhao] for the
> [discussions|https://issues.apache.org/jira/browse/HDFS-8786?focusedCommentId=15180415=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15180415]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219208#comment-15219208
]
Vinayakumar B commented on HDFS-9263:
-
Hi [~ste...@apache.org],
As of current jenkins reports seen, there is no such files created outside
"target" directory which could not be cleaned by "mvn clean".
If such files present jenkins would have caught it under "ASF LICENCE
warnings".
Still you want this to be fixed before 2.7.3 ?
Patch does a nice a Job of centralizing the usage of system property
"test.build.data".
But seeing the intention of this Jira, and current state of jenkins reports, I
feel think this might not be a blocker for 2.7.3.
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8496) Calling stopWriter() with FSDatasetImpl lock held may block other threads
[
https://issues.apache.org/jira/browse/HDFS-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219196#comment-15219196
]
Hadoop QA commented on HDFS-8496:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 4 new +
140 unchanged - 3 fixed = 144 total (was 143) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 14s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 16s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 25s
{color} | {color:red} Patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 184m 29s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests | hadoop.hdfs.TestFileCreationDelete |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.TestDataTransferProtocol |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.namenode.TestINodeFile |
| | hadoop.hdfs.TestReadWhileWriting |
| |
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[
https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219169#comment-15219169
]
Lin Yiqun commented on HDFS-9847:
-
Hi, [~chris.douglas], the latest v003 patch has fix these errors. The latest
jenkins report's failed unit tests is not related to this JIRA. I also agree on
this:
{quote}
IMO, we should go ahead without changing the default values and breaking tests
in this Jira.
{quote}
We can just update hdfs-default.xml without changing the default values in v003
patch.
> HDFS configuration without time unit name should accept friendly time units
> ---
>
> Key: HDFS-9847
> URL: https://issues.apache.org/jira/browse/HDFS-9847
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HDFS-9847-branch-2.001.patch,
> HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch,
> HDFS-9847-nothrow.002.patch, HDFS-9847-nothrow.003.patch,
> HDFS-9847.001.patch, HDFS-9847.002.patch, HDFS-9847.003.patch,
> HDFS-9847.004.patch, HDFS-9847.005.patch, HDFS-9847.006.patch,
> branch-2-delta.002.txt, timeduration-w-y.patch
>
>
> In HDFS-9821, it talks about the issue of leting existing keys use friendly
> units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names
> contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make
> some other configurations which without time unit name to accept friendly
> time units. The time unit {{seconds}} is frequently used in hdfs. We can
> updating this configurations first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9943) Support reconfiguring namenode replication confs
[ https://issues.apache.org/jira/browse/HDFS-9943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-9943: Attachment: HDFS-9943-HDFS-9000.000.patch > Support reconfiguring namenode replication confs > > > Key: HDFS-9943 > URL: https://issues.apache.org/jira/browse/HDFS-9943 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Xiaobing Zhou > Attachments: HDFS-9943-HDFS-9000.000.patch > > > The following confs should be re-configurable in runtime. > - dfs.namenode.replication.work.multiplier.per.iteration > - dfs.namenode.replication.interval > - dfs.namenode.replication.max-streams > - dfs.namenode.replication.max-streams-hard-limit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9943) Support reconfiguring namenode replication confs
[ https://issues.apache.org/jira/browse/HDFS-9943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219149#comment-15219149 ] Xiaobing Zhou commented on HDFS-9943: - Here's the patch v000 for review, unit tests will be added in next patch. > Support reconfiguring namenode replication confs > > > Key: HDFS-9943 > URL: https://issues.apache.org/jira/browse/HDFS-9943 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Xiaobing Zhou > Attachments: HDFS-9943-HDFS-9000.000.patch > > > The following confs should be re-configurable in runtime. > - dfs.namenode.replication.work.multiplier.per.iteration > - dfs.namenode.replication.interval > - dfs.namenode.replication.max-streams > - dfs.namenode.replication.max-streams-hard-limit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9943) Support reconfiguring namenode replication confs
[ https://issues.apache.org/jira/browse/HDFS-9943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-9943: Status: Patch Available (was: Open) > Support reconfiguring namenode replication confs > > > Key: HDFS-9943 > URL: https://issues.apache.org/jira/browse/HDFS-9943 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Xiaobing Zhou > Attachments: HDFS-9943-HDFS-9000.000.patch > > > The following confs should be re-configurable in runtime. > - dfs.namenode.replication.work.multiplier.per.iteration > - dfs.namenode.replication.interval > - dfs.namenode.replication.max-streams > - dfs.namenode.replication.max-streams-hard-limit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219123#comment-15219123
]
Vinod Kumar Vavilapalli commented on HDFS-9263:
---
Either ways, can you please see if this can get into 2.7.3 within a week? Tx.
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8674) Improve performance of postponed block scans
[ https://issues.apache.org/jira/browse/HDFS-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219121#comment-15219121 ] Vinod Kumar Vavilapalli commented on HDFS-8674: --- [~daryn] / [~mingma], is it possible to get this in 2.7.3 in a week? Tx? > Improve performance of postponed block scans > > > Key: HDFS-8674 > URL: https://issues.apache.org/jira/browse/HDFS-8674 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.6.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HDFS-8674.patch, HDFS-8674.patch > > > When a standby goes active, it marks all nodes as "stale" which will cause > block invalidations for over-replicated blocks to be queued until full block > reports are received from the nodes with the block. The replication monitor > scans the queue with O(N) runtime. It picks a random offset and iterates > through the set to randomize blocks scanned. > The result is devastating when a cluster loses multiple nodes during a > rolling upgrade. Re-replication occurs, the nodes come back, the excess block > invalidations are postponed. Rescanning just 2k blocks out of millions of > postponed blocks may take multiple seconds. During the scan, the write lock > is held which stalls all other processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10221) Add .json to the rat exclusions
[ https://issues.apache.org/jira/browse/HDFS-10221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219118#comment-15219118 ] Ming Ma commented on HDFS-10221: Thanks all! > Add .json to the rat exclusions > --- > > Key: HDFS-10221 > URL: https://issues.apache.org/jira/browse/HDFS-10221 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 2.8.0 >Reporter: Ming Ma >Assignee: Ming Ma >Priority: Blocker > Fix For: 2.8.0 > > Attachments: HDFS-10221-2.patch, HDFS-10221.patch > > > A new test resource dfs.hosts.json was added to HDFS-9005 for better > readability. Given json file doesn't allow comments, it violates ASF license. > To address this, we can add the file to rat exclusions list in the pom.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10221) Add .json to the rat exclusions
[ https://issues.apache.org/jira/browse/HDFS-10221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219102#comment-15219102 ] Hudson commented on HDFS-10221: --- FAILURE: Integrated in Hadoop-trunk-Commit #9530 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9530/]) HDFS-10221. Add .json to the rat exclusions. Contributed by Ming Ma. (aajisaka: rev 32c0c3ecdf72e89a63f4aee5e75d1c5a12714b89) * hadoop-hdfs-project/hadoop-hdfs/pom.xml > Add .json to the rat exclusions > --- > > Key: HDFS-10221 > URL: https://issues.apache.org/jira/browse/HDFS-10221 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 2.8.0 >Reporter: Ming Ma >Assignee: Ming Ma >Priority: Blocker > Fix For: 2.8.0 > > Attachments: HDFS-10221-2.patch, HDFS-10221.patch > > > A new test resource dfs.hosts.json was added to HDFS-9005 for better > readability. Given json file doesn't allow comments, it violates ASF license. > To address this, we can add the file to rat exclusions list in the pom.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10221) Add .json to the rat exclusions
[ https://issues.apache.org/jira/browse/HDFS-10221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-10221: - Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Committed v2 patch to trunk, branch-2, and branch-2.8. Thanks all for contributing to this issue. > Add .json to the rat exclusions > --- > > Key: HDFS-10221 > URL: https://issues.apache.org/jira/browse/HDFS-10221 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 2.8.0 >Reporter: Ming Ma >Assignee: Ming Ma >Priority: Blocker > Fix For: 2.8.0 > > Attachments: HDFS-10221-2.patch, HDFS-10221.patch > > > A new test resource dfs.hosts.json was added to HDFS-9005 for better > readability. Given json file doesn't allow comments, it violates ASF license. > To address this, we can add the file to rat exclusions list in the pom.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10221) Add .json to the rat exclusions
[ https://issues.apache.org/jira/browse/HDFS-10221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-10221: - Target Version/s: 2.8.0 Environment: (was: Jenkins) Hadoop Flags: Reviewed Summary: Add .json to the rat exclusions (was: Add test resource dfs.hosts.json to the rat exclusions) > Add .json to the rat exclusions > --- > > Key: HDFS-10221 > URL: https://issues.apache.org/jira/browse/HDFS-10221 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 2.8.0 >Reporter: Ming Ma >Assignee: Ming Ma >Priority: Blocker > Attachments: HDFS-10221-2.patch, HDFS-10221.patch > > > A new test resource dfs.hosts.json was added to HDFS-9005 for better > readability. Given json file doesn't allow comments, it violates ASF license. > To address this, we can add the file to rat exclusions list in the pom.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8498) Blocks can be committed with wrong size
[ https://issues.apache.org/jira/browse/HDFS-8498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219079#comment-15219079 ] Zhe Zhang commented on HDFS-8498: - I'm OK with closing as won't fix, if Daryn hasn't seen new occurrences of the bug after HDFS-9289. > Blocks can be committed with wrong size > --- > > Key: HDFS-8498 > URL: https://issues.apache.org/jira/browse/HDFS-8498 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.5.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > > When an IBR for a UC block arrives, the NN updates the expected location's > block and replica state _only_ if it's on an unexpected storage for an > expected DN. If it's for an expected storage, only the genstamp is updated. > When the block is committed, and the expected locations are verified, only > the genstamp is checked. The size is not checked but it wasn't updated in > the expected locations anyway. > A faulty client may misreport the size when committing the block. The block > is effectively corrupted. If the NN issues replications, the received IBR is > considered corrupt, the NN invalidates the block, immediately issues another > replication. The NN eventually realizes all the original replicas are > corrupt after full BRs are received from the original DNs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10221) Add test resource dfs.hosts.json to the rat exclusions
[ https://issues.apache.org/jira/browse/HDFS-10221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219069#comment-15219069 ] Akira AJISAKA commented on HDFS-10221: -- +1 > Add test resource dfs.hosts.json to the rat exclusions > -- > > Key: HDFS-10221 > URL: https://issues.apache.org/jira/browse/HDFS-10221 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 2.8.0 > Environment: Jenkins >Reporter: Ming Ma >Assignee: Ming Ma >Priority: Blocker > Attachments: HDFS-10221-2.patch, HDFS-10221.patch > > > A new test resource dfs.hosts.json was added to HDFS-9005 for better > readability. Given json file doesn't allow comments, it violates ASF license. > To address this, we can add the file to rat exclusions list in the pom.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10238) Ozone : Add chunk persistance
[ https://issues.apache.org/jira/browse/HDFS-10238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-10238: Status: Patch Available (was: Open) > Ozone : Add chunk persistance > - > > Key: HDFS-10238 > URL: https://issues.apache.org/jira/browse/HDFS-10238 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-10238-HDFS-7240.001.patch > > > Add support for chunks in containers for ozone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10238) Ozone : Add chunk persistance
[ https://issues.apache.org/jira/browse/HDFS-10238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-10238: Attachment: HDFS-10238-HDFS-7240.001.patch > Ozone : Add chunk persistance > - > > Key: HDFS-10238 > URL: https://issues.apache.org/jira/browse/HDFS-10238 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-10238-HDFS-7240.001.patch > > > Add support for chunks in containers for ozone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8498) Blocks can be committed with wrong size
[ https://issues.apache.org/jira/browse/HDFS-8498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219057#comment-15219057 ] Vinod Kumar Vavilapalli commented on HDFS-8498: --- [~zhz] / [~kihwal] / [~daryn], I think we should close this as Won't fix or as a dup of HDFS-9289. This bug keeps appearing in the blocker/critical list for releases, but we don't seem to be progressing. > Blocks can be committed with wrong size > --- > > Key: HDFS-8498 > URL: https://issues.apache.org/jira/browse/HDFS-8498 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.5.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > > When an IBR for a UC block arrives, the NN updates the expected location's > block and replica state _only_ if it's on an unexpected storage for an > expected DN. If it's for an expected storage, only the genstamp is updated. > When the block is committed, and the expected locations are verified, only > the genstamp is checked. The size is not checked but it wasn't updated in > the expected locations anyway. > A faulty client may misreport the size when committing the block. The block > is effectively corrupted. If the NN issues replications, the received IBR is > considered corrupt, the NN invalidates the block, immediately issues another > replication. The NN eventually realizes all the original replicas are > corrupt after full BRs are received from the original DNs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8496) Calling stopWriter() with FSDatasetImpl lock held may block other threads
[
https://issues.apache.org/jira/browse/HDFS-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-8496:
--
Target Version/s: 2.7.3, 2.6.5
> Calling stopWriter() with FSDatasetImpl lock held may block other threads
> -
>
> Key: HDFS-8496
> URL: https://issues.apache.org/jira/browse/HDFS-8496
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: zhouyingchao
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-8496-001.patch, HDFS-8496.002.patch
>
>
> On a DN of a HDFS 2.6 cluster, we noticed some DataXceiver threads and
> heartbeat threads are blocked for quite a while on the FSDatasetImpl lock. By
> looking at the stack, we found the calling of stopWriter() with FSDatasetImpl
> lock blocked everything.
> Following is the heartbeat stack, as an example, to show how threads are
> blocked by FSDatasetImpl lock:
> {code}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getDfsUsed(FsVolumeImpl.java:152)
> - waiting to lock <0x0007701badc0> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getAvailable(FsVolumeImpl.java:191)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getStorageReports(FsDatasetImpl.java:144)
> - locked <0x000770465dc0> (a java.lang.Object)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:575)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:680)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:850)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> The thread which held the FSDatasetImpl lock is just sleeping to wait another
> thread to exit in stopWriter(). The stack is:
> {code}
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1194)
> - locked <0x0007636953b8> (a org.apache.hadoop.util.Daemon)
> at
> org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.stopWriter(ReplicaInPipeline.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.recoverCheck(FsDatasetImpl.java:982)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.recoverClose(FsDatasetImpl.java:1026)
> - locked <0x0007701badc0> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:624)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> In this case, we deployed quite a lot other workloads on the DN, the local
> file system and disk is quite busy. We guess this is why the stopWriter took
> quite a long time.
> Any way, it is not quite reasonable to call stopWriter with the FSDatasetImpl
> lock held. In HDFS-7999, the createTemporary() is changed to call
> stopWriter without FSDatasetImpl lock. We guess we should do so in the other
> three methods: recoverClose()/recoverAppend/recoverRbw().
> I'll try to finish a patch for this today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9908) Datanode should tolerate disk scan failure during NN handshake
[
https://issues.apache.org/jira/browse/HDFS-9908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219013#comment-15219013
]
Hadoop QA commented on HDFS-9908:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 3s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 53s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 46s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 49s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 55s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 48s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 16s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 53s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 8s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 27s
{color} | {color:red} Patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 215m 5s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeMetrics |
|
[jira] [Assigned] (HDFS-8496) Calling stopWriter() with FSDatasetImpl lock held may block other threads
[
https://issues.apache.org/jira/browse/HDFS-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe reassigned HDFS-8496:
--
Assignee: Colin Patrick McCabe (was: zhouyingchao)
> Calling stopWriter() with FSDatasetImpl lock held may block other threads
> -
>
> Key: HDFS-8496
> URL: https://issues.apache.org/jira/browse/HDFS-8496
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: zhouyingchao
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-8496-001.patch, HDFS-8496.002.patch
>
>
> On a DN of a HDFS 2.6 cluster, we noticed some DataXceiver threads and
> heartbeat threads are blocked for quite a while on the FSDatasetImpl lock. By
> looking at the stack, we found the calling of stopWriter() with FSDatasetImpl
> lock blocked everything.
> Following is the heartbeat stack, as an example, to show how threads are
> blocked by FSDatasetImpl lock:
> {code}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getDfsUsed(FsVolumeImpl.java:152)
> - waiting to lock <0x0007701badc0> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getAvailable(FsVolumeImpl.java:191)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getStorageReports(FsDatasetImpl.java:144)
> - locked <0x000770465dc0> (a java.lang.Object)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:575)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:680)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:850)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> The thread which held the FSDatasetImpl lock is just sleeping to wait another
> thread to exit in stopWriter(). The stack is:
> {code}
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1194)
> - locked <0x0007636953b8> (a org.apache.hadoop.util.Daemon)
> at
> org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.stopWriter(ReplicaInPipeline.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.recoverCheck(FsDatasetImpl.java:982)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.recoverClose(FsDatasetImpl.java:1026)
> - locked <0x0007701badc0> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:624)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> In this case, we deployed quite a lot other workloads on the DN, the local
> file system and disk is quite busy. We guess this is why the stopWriter took
> quite a long time.
> Any way, it is not quite reasonable to call stopWriter with the FSDatasetImpl
> lock held. In HDFS-7999, the createTemporary() is changed to call
> stopWriter without FSDatasetImpl lock. We guess we should do so in the other
> three methods: recoverClose()/recoverAppend/recoverRbw().
> I'll try to finish a patch for this today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8496) Calling stopWriter() with FSDatasetImpl lock held may block other threads
[
https://issues.apache.org/jira/browse/HDFS-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218998#comment-15218998
]
Colin Patrick McCabe commented on HDFS-8496:
Hi [~sinago],
Sorry for the delay in reviews. I have rebased this patch on the current trunk
(and made a few changes), mind taking a look?
> Calling stopWriter() with FSDatasetImpl lock held may block other threads
> -
>
> Key: HDFS-8496
> URL: https://issues.apache.org/jira/browse/HDFS-8496
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: zhouyingchao
>Assignee: zhouyingchao
> Attachments: HDFS-8496-001.patch, HDFS-8496.002.patch
>
>
> On a DN of a HDFS 2.6 cluster, we noticed some DataXceiver threads and
> heartbeat threads are blocked for quite a while on the FSDatasetImpl lock. By
> looking at the stack, we found the calling of stopWriter() with FSDatasetImpl
> lock blocked everything.
> Following is the heartbeat stack, as an example, to show how threads are
> blocked by FSDatasetImpl lock:
> {code}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getDfsUsed(FsVolumeImpl.java:152)
> - waiting to lock <0x0007701badc0> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getAvailable(FsVolumeImpl.java:191)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getStorageReports(FsDatasetImpl.java:144)
> - locked <0x000770465dc0> (a java.lang.Object)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:575)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:680)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:850)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> The thread which held the FSDatasetImpl lock is just sleeping to wait another
> thread to exit in stopWriter(). The stack is:
> {code}
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1194)
> - locked <0x0007636953b8> (a org.apache.hadoop.util.Daemon)
> at
> org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.stopWriter(ReplicaInPipeline.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.recoverCheck(FsDatasetImpl.java:982)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.recoverClose(FsDatasetImpl.java:1026)
> - locked <0x0007701badc0> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:624)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> In this case, we deployed quite a lot other workloads on the DN, the local
> file system and disk is quite busy. We guess this is why the stopWriter took
> quite a long time.
> Any way, it is not quite reasonable to call stopWriter with the FSDatasetImpl
> lock held. In HDFS-7999, the createTemporary() is changed to call
> stopWriter without FSDatasetImpl lock. We guess we should do so in the other
> three methods: recoverClose()/recoverAppend/recoverRbw().
> I'll try to finish a patch for this today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8496) Calling stopWriter() with FSDatasetImpl lock held may block other threads
[
https://issues.apache.org/jira/browse/HDFS-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-8496:
---
Attachment: HDFS-8496.002.patch
> Calling stopWriter() with FSDatasetImpl lock held may block other threads
> -
>
> Key: HDFS-8496
> URL: https://issues.apache.org/jira/browse/HDFS-8496
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: zhouyingchao
>Assignee: zhouyingchao
> Attachments: HDFS-8496-001.patch, HDFS-8496.002.patch
>
>
> On a DN of a HDFS 2.6 cluster, we noticed some DataXceiver threads and
> heartbeat threads are blocked for quite a while on the FSDatasetImpl lock. By
> looking at the stack, we found the calling of stopWriter() with FSDatasetImpl
> lock blocked everything.
> Following is the heartbeat stack, as an example, to show how threads are
> blocked by FSDatasetImpl lock:
> {code}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getDfsUsed(FsVolumeImpl.java:152)
> - waiting to lock <0x0007701badc0> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getAvailable(FsVolumeImpl.java:191)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getStorageReports(FsDatasetImpl.java:144)
> - locked <0x000770465dc0> (a java.lang.Object)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:575)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:680)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:850)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> The thread which held the FSDatasetImpl lock is just sleeping to wait another
> thread to exit in stopWriter(). The stack is:
> {code}
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1194)
> - locked <0x0007636953b8> (a org.apache.hadoop.util.Daemon)
> at
> org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.stopWriter(ReplicaInPipeline.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.recoverCheck(FsDatasetImpl.java:982)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.recoverClose(FsDatasetImpl.java:1026)
> - locked <0x0007701badc0> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:624)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> In this case, we deployed quite a lot other workloads on the DN, the local
> file system and disk is quite busy. We guess this is why the stopWriter took
> quite a long time.
> Any way, it is not quite reasonable to call stopWriter with the FSDatasetImpl
> lock held. In HDFS-7999, the createTemporary() is changed to call
> stopWriter without FSDatasetImpl lock. We guess we should do so in the other
> three methods: recoverClose()/recoverAppend/recoverRbw().
> I'll try to finish a patch for this today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-10238) Ozone : Add chunk persistance
Anu Engineer created HDFS-10238: --- Summary: Ozone : Add chunk persistance Key: HDFS-10238 URL: https://issues.apache.org/jira/browse/HDFS-10238 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: HDFS-7240 Add support for chunks in containers for ozone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9918) Erasure Coding: Sort located striped blocks based on decommissioned states
[
https://issues.apache.org/jira/browse/HDFS-9918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218965#comment-15218965
]
Zhe Zhang commented on HDFS-9918:
-
In my original comment when I said preserving index ordering, I meant something
like below (using Rakesh's example):
bq. Assume in the below list 0, 1, 7, 8 is decommissioned and extra redundant
index is 1.
bq. 0, 1', 2, 3, 4, 5, 6, 7', 8', 0', 1, 7, 8, 1
In the above, I marked decommissioning internal blocks with a {{'}}. I think we
should order the list as:
{code}
0, 1, 2, 3, 4, 5, 6, 7, 8, 1, 0', 1', 7', 8'
{code}
The main motivation is to enable future optimization HDFS-8032. So actually I
think we should do a decomm-based sorting within a block index, and only "swap"
2 internal blocks with the same block index.
In the v8 patch, I think we can use a {{DECOM_COMPARATOR}}-based sort on each
list in {{blkIndex2LocationsMap}}. Then for the first {{numDataUnits}} slots in
the result, we take the first location from the index. From sorting priority's
perspective, I think a decommissioning internal block is still more useful then
a redundant internal block.
LMK if you guys agree.
> Erasure Coding: Sort located striped blocks based on decommissioned states
> --
>
> Key: HDFS-9918
> URL: https://issues.apache.org/jira/browse/HDFS-9918
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-9918-001.patch, HDFS-9918-002.patch,
> HDFS-9918-003.patch, HDFS-9918-004.patch, HDFS-9918-005.patch,
> HDFS-9918-006.patch, HDFS-9918-007.patch, HDFS-9918-008.patch
>
>
> This jira is a follow-on work of HDFS-8786, where we do decommissioning of
> datanodes having striped blocks.
> Now, after decommissioning it requires to change the ordering of the storage
> list so that the decommissioned datanodes should only be last node in list.
> For example, assume we have a block group with storage list:-
> d0, d1, d2, d3, d4, d5, d6, d7, d8, d9
> mapping to indices
> 0, 1, 2, 3, 4, 5, 6, 7, 8, 2
> Here the internal block b2 is duplicated, locating in d2 and d9. If d2 is a
> decommissioning node then should switch d2 and d9 in the storage list.
> Thanks [~jingzhao] for the
> [discussions|https://issues.apache.org/jira/browse/HDFS-8786?focusedCommentId=15180415=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15180415]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5280) Corrupted meta files on data nodes prevents DFClient from connecting to data nodes and updating corruption status to name node.
[
https://issues.apache.org/jira/browse/HDFS-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218951#comment-15218951
]
Andres Perez commented on HDFS-5280:
The problem is happening because the {{IllegalArgumentException}} is not being
handled at any level. Maybe the solution might be throwing a
{{ChecksumException}} also in this case, or create a new type of exception
indicating the wrong format of the blk*.meta file.
At some point as well the affected block should be marked as corrupt.
> Corrupted meta files on data nodes prevents DFClient from connecting to data
> nodes and updating corruption status to name node.
> ---
>
> Key: HDFS-5280
> URL: https://issues.apache.org/jira/browse/HDFS-5280
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, hdfs-client
>Affects Versions: 1.1.1, 2.1.0-beta, 2.0.4-alpha, 2.7.2
> Environment: Red hat enterprise 6.4
> Hadoop-2.1.0
>Reporter: Jinghui Wang
>Assignee: Andres Perez
>
> Meta files being corrupted causes the DFSClient not able to connect to the
> datanodes to access the blocks, so DFSClient never perform a read on the
> block, which is what throws the ChecksumException when file blocks are
> corrupted and report to the namenode to mark the block as corrupt. Since the
> client never got to that far, thus the file status remain as healthy and so
> are all the blocks.
> To replicate the error, put a file onto HDFS.
> run hadoop fsck /tmp/bogus.csv -files -blocks -location will get that
> following output.
> FSCK started for path /tmp/bogus.csv at 11:33:29
> /tmp/bogus.csv 109 bytes, 1 block(s): OK
> 0. blk_-4255166695856420554_5292 len=109 repl=3
> find the block/meta files for 4255166695856420554 by running
> ssh datanode1.address find /hadoop/ -name "*4255166695856420554*" and it will
> get the following output:
> /hadoop/data1/hdfs/current/subdir2/blk_-4255166695856420554
> /hadoop/data1/hdfs/current/subdir2/blk_-4255166695856420554_5292.meta
> now corrupt the meta file by running
> ssh datanode1.address "sed -i -e '1i 1234567891'
> /hadoop/data1/hdfs/current/subdir2/blk_-4255166695856420554_5292.meta"
> now run hadoop fs -cat /tmp/bogus.csv
> will show the stack trace of DFSClient failing to connect to the data node
> with the corrupted meta file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7240) Object store in HDFS
[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-7240: Assignee: Jitendra Nath Pandey (was: Chris Nauroth) > Object store in HDFS > > > Key: HDFS-7240 > URL: https://issues.apache.org/jira/browse/HDFS-7240 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: Ozone-architecture-v1.pdf > > > This jira proposes to add object store capabilities into HDFS. > As part of the federation work (HDFS-1052) we separated block storage as a > generic storage layer. Using the Block Pool abstraction, new kinds of > namespaces can be built on top of the storage layer i.e. datanodes. > In this jira I will explore building an object store using the datanode > storage, but independent of namespace metadata. > I will soon update with a detailed design document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8210) Ozone: Implement storage container manager
[ https://issues.apache.org/jira/browse/HDFS-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-8210: --- Resolution: Fixed Status: Resolved (was: Patch Available) [~cnauroth] Thanks for the patch. I have committed this to HDFS-7240. > Ozone: Implement storage container manager > --- > > Key: HDFS-8210 > URL: https://issues.apache.org/jira/browse/HDFS-8210 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Jitendra Nath Pandey >Assignee: Chris Nauroth > Attachments: HDFS-8210-HDFS-7240.006.patch, > HDFS-8210-HDFS-7240.007.patch, HDFS-8210-HDFS-7240.1.patch, > HDFS-8210-HDFS-7240.2.patch, HDFS-8210-HDFS-7240.3.patch, > HDFS-8210-HDFS-7240.4.patch, HDFS-8210-HDFS-7240.5.patch > > > The storage container manager collects datanode heartbeats, manages > replication and exposes API to lookup containers. This jira implements > storage container manager by re-using the block manager implementation in > namenode. This jira provides initial implementation that works with > datanodes. The additional protocols will be added in subsequent jiras. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HDFS-7240) Object store in HDFS
[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer reopened HDFS-7240: > Object store in HDFS > > > Key: HDFS-7240 > URL: https://issues.apache.org/jira/browse/HDFS-7240 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Chris Nauroth > Attachments: Ozone-architecture-v1.pdf > > > This jira proposes to add object store capabilities into HDFS. > As part of the federation work (HDFS-1052) we separated block storage as a > generic storage layer. Using the Block Pool abstraction, new kinds of > namespaces can be built on top of the storage layer i.e. datanodes. > In this jira I will explore building an object store using the datanode > storage, but independent of namespace metadata. > I will soon update with a detailed design document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HDFS-7240) Object store in HDFS
[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-7240: Comment: was deleted (was: [~cnauroth] Thanks for the patch. I have committed this to HDFS-7240.) > Object store in HDFS > > > Key: HDFS-7240 > URL: https://issues.apache.org/jira/browse/HDFS-7240 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Chris Nauroth > Attachments: Ozone-architecture-v1.pdf > > > This jira proposes to add object store capabilities into HDFS. > As part of the federation work (HDFS-1052) we separated block storage as a > generic storage layer. Using the Block Pool abstraction, new kinds of > namespaces can be built on top of the storage layer i.e. datanodes. > In this jira I will explore building an object store using the datanode > storage, but independent of namespace metadata. > I will soon update with a detailed design document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7240) Object store in HDFS
[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDFS-7240. Resolution: Fixed Assignee: Chris Nauroth (was: Jitendra Nath Pandey) Hadoop Flags: Reviewed [~cnauroth] Thanks for the patch. I have committed this to HDFS-7240. > Object store in HDFS > > > Key: HDFS-7240 > URL: https://issues.apache.org/jira/browse/HDFS-7240 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Chris Nauroth > Attachments: Ozone-architecture-v1.pdf > > > This jira proposes to add object store capabilities into HDFS. > As part of the federation work (HDFS-1052) we separated block storage as a > generic storage layer. Using the Block Pool abstraction, new kinds of > namespaces can be built on top of the storage layer i.e. datanodes. > In this jira I will explore building an object store using the datanode > storage, but independent of namespace metadata. > I will soon update with a detailed design document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8210) Ozone: Implement storage container manager
[
https://issues.apache.org/jira/browse/HDFS-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218921#comment-15218921
]
Hadoop QA commented on HDFS-8210:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
30s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 18s
{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
54s {color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 19s
{color} | {color:green} HDFS-7240 passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 52s
{color} | {color:green} HDFS-7240 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 25s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 44s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 109m 58s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 49s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 23s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 238m 46s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests | hadoop.hdfs.server.namenode.TestCheckpoint
|
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestPersistBlocks |
| |
[jira] [Commented] (HDFS-9890) libhdfs++: Add test suite to simulate network issues
[
https://issues.apache.org/jira/browse/HDFS-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218885#comment-15218885
]
Hadoop QA commented on HDFS-9890:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 46s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
47s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 2s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 8s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s
{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 34m 4s {color} |
{color:red} hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_77 with JDK
v1.8.0_77 generated 41 new + 29 unchanged - 0 fixed = 70 total (was 29) {color}
|
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 28s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 40m 34s {color} |
{color:red} hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.7.0_95 with JDK
v1.7.0_95 generated 41 new + 29 unchanged - 0 fixed = 70 total (was 29) {color}
|
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 19s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 56s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 24s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12796128/HDFS-9890.HDFS-8707.000.patch
|
| JIRA Issue | HDFS-9890 |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 4476de2257d6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / c2e1e23 |
| Default Java | 1.7.0_95 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_77
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 |
| cc | hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_77:
[jira] [Commented] (HDFS-10207) Support enable Hadoop IPC backoff without namenode restart
[
https://issues.apache.org/jira/browse/HDFS-10207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218864#comment-15218864
]
Hadoop QA commented on HDFS-10207:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HDFS-10207 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12796157/HDFS-10207-HDFS-9000.001.patch
|
| JIRA Issue | HDFS-10207 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15008/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Support enable Hadoop IPC backoff without namenode restart
> --
>
> Key: HDFS-10207
> URL: https://issues.apache.org/jira/browse/HDFS-10207
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10207-HDFS-9000.000.patch,
> HDFS-10207-HDFS-9000.001.patch
>
>
> It will be useful to allow changing {{ipc.#port#.backoff.enable}} without a
> namenode restart to protect namenode from being overloaded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10223) peerFromSocketAndKey performs SASL exchange before setting connection timeouts
[
https://issues.apache.org/jira/browse/HDFS-10223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218863#comment-15218863
]
Hudson commented on HDFS-10223:
---
FAILURE: Integrated in Hadoop-trunk-Commit #9528 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9528/])
HDFS-10223. peerFromSocketAndKey performs SASL exchange before setting
(cmccabe: rev 37e23ce45c592f3c9c48a08a52a5f46787f6c0e9)
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/TestSaslDataTransfer.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
> peerFromSocketAndKey performs SASL exchange before setting connection timeouts
> --
>
> Key: HDFS-10223
> URL: https://issues.apache.org/jira/browse/HDFS-10223
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.4
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.8.0
>
> Attachments: HDFS-10223.001.patch, HDFS-10223.002.patch
>
>
> {{peerFromSocketAndKey}} performs the SASL exchange before setting up
> connection timeouts. Because of this, the timeout used for setting up SASL
> connections is the default system-wide TCP timeout, which is usually several
> hours long.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9869) Erasure Coding: Rename replication-based names in BlockManager to more generic [part-2]
[
https://issues.apache.org/jira/browse/HDFS-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218847#comment-15218847
]
Hadoop QA commented on HDFS-9869:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 22 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
26s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
32s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 2s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 27s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 18s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 11s
{color} | {color:red} root-jdk1.8.0_74 with JDK v1.8.0_74 generated 2 new + 736
unchanged - 2 fixed = 738 total (was 738) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 17m 32s
{color} | {color:red} root-jdk1.7.0_95 with JDK v1.7.0_95 generated 2 new + 733
unchanged - 2 fixed = 735 total (was 735) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 21s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 11s
{color} | {color:red} root: patch generated 2 new + 747 unchanged - 3 fixed =
749 total (was 750) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 58s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 22s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 10s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 22s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} |
[jira] [Updated] (HDFS-10209) Support enable caller context in HDFS namenode audit log without restart namenode
[ https://issues.apache.org/jira/browse/HDFS-10209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10209: - Attachment: HDFS-10209-HDFS-9000.000.patch re-posted the patch to trigger the jenkins run. > Support enable caller context in HDFS namenode audit log without restart > namenode > - > > Key: HDFS-10209 > URL: https://issues.apache.org/jira/browse/HDFS-10209 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Xiaobing Zhou > Attachments: HDFS-10209-HDFS-9000.000.patch > > > RPC caller context is a useful feature to track down the origin of the > caller, which can track down "bad" jobs that overload the namenode. This > ticket is opened to allow enabling caller context without namenode restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10231) libhdfs++: Fix race conditions in RPC layer
[
https://issues.apache.org/jira/browse/HDFS-10231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218840#comment-15218840
]
Hadoop QA commented on HDFS-10231:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 33s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
33s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 41s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 44s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s
{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 28s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 28s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 52s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12796147/HDFS-10231.HDFS-8707.002.patch
|
| JIRA Issue | HDFS-10231 |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux b465d7004504 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / c2e1e23 |
| Default Java | 1.7.0_95 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_77
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 |
| JDK v1.7.0_95 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15006/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15006/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> libhdfs++: Fix race conditions in RPC layer
> ---
>
> Key: HDFS-10231
> URL: https://issues.apache.org/jira/browse/HDFS-10231
> Project: Hadoop HDFS
> Issue Type: Sub-task
[jira] [Updated] (HDFS-10209) Support enable caller context in HDFS namenode audit log without restart namenode
[ https://issues.apache.org/jira/browse/HDFS-10209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10209: - Attachment: (was: HDFS-10209-HDFS-9000.000.patch) > Support enable caller context in HDFS namenode audit log without restart > namenode > - > > Key: HDFS-10209 > URL: https://issues.apache.org/jira/browse/HDFS-10209 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Xiaobing Zhou > > RPC caller context is a useful feature to track down the origin of the > caller, which can track down "bad" jobs that overload the namenode. This > ticket is opened to allow enabling caller context without namenode restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10207) Support enable Hadoop IPC backoff without namenode restart
[
https://issues.apache.org/jira/browse/HDFS-10207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HDFS-10207:
-
Status: Patch Available (was: Open)
> Support enable Hadoop IPC backoff without namenode restart
> --
>
> Key: HDFS-10207
> URL: https://issues.apache.org/jira/browse/HDFS-10207
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10207-HDFS-9000.000.patch,
> HDFS-10207-HDFS-9000.001.patch
>
>
> It will be useful to allow changing {{ipc.#port#.backoff.enable}} without a
> namenode restart to protect namenode from being overloaded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-10207) Support enable Hadoop IPC backoff without namenode restart
[
https://issues.apache.org/jira/browse/HDFS-10207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HDFS-10207:
-
Attachment: HDFS-10207-HDFS-9000.001.patch
Here's the patch v001, added some unit tests.
> Support enable Hadoop IPC backoff without namenode restart
> --
>
> Key: HDFS-10207
> URL: https://issues.apache.org/jira/browse/HDFS-10207
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10207-HDFS-9000.000.patch,
> HDFS-10207-HDFS-9000.001.patch
>
>
> It will be useful to allow changing {{ipc.#port#.backoff.enable}} without a
> namenode restart to protect namenode from being overloaded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-10223) peerFromSocketAndKey performs SASL exchange before setting connection timeouts
[
https://issues.apache.org/jira/browse/HDFS-10223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-10223:
Resolution: Fixed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
> peerFromSocketAndKey performs SASL exchange before setting connection timeouts
> --
>
> Key: HDFS-10223
> URL: https://issues.apache.org/jira/browse/HDFS-10223
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.4
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.8.0
>
> Attachments: HDFS-10223.001.patch, HDFS-10223.002.patch
>
>
> {{peerFromSocketAndKey}} performs the SASL exchange before setting up
> connection timeouts. Because of this, the timeout used for setting up SASL
> connections is the default system-wide TCP timeout, which is usually several
> hours long.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10223) peerFromSocketAndKey performs SASL exchange before setting connection timeouts
[
https://issues.apache.org/jira/browse/HDFS-10223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218816#comment-15218816
]
Colin Patrick McCabe commented on HDFS-10223:
-
Finally got a jenkins run. Committed to 2.8. Thanks, all.
> peerFromSocketAndKey performs SASL exchange before setting connection timeouts
> --
>
> Key: HDFS-10223
> URL: https://issues.apache.org/jira/browse/HDFS-10223
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.4
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-10223.001.patch, HDFS-10223.002.patch
>
>
> {{peerFromSocketAndKey}} performs the SASL exchange before setting up
> connection timeouts. Because of this, the timeout used for setting up SASL
> connections is the default system-wide TCP timeout, which is usually several
> hours long.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9890) libhdfs++: Add test suite to simulate network issues
[ https://issues.apache.org/jira/browse/HDFS-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9890: -- Status: Patch Available (was: Open) > libhdfs++: Add test suite to simulate network issues > > > Key: HDFS-9890 > URL: https://issues.apache.org/jira/browse/HDFS-9890 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Stephen > Attachments: HDFS-9890.HDFS-8707.000.patch > > > I propose adding a test suite to simulate various network issues/failures in > order to get good test coverage on some of the retry paths that aren't easy > to hit in mock unit tests. > At the moment the only things that hit the retry paths are the gmock unit > tests. The gmock are only as good as their mock implementations which do a > great job of simulating protocol correctness but not more complex > interactions. They also can't really simulate the types of lock contention > and subtle memory stomps that show up while doing hundreds or thousands of > concurrent reads. We should add a new minidfscluster test that focuses on > heavy read/seek load and then randomly convert error codes returned by > network functions into errors. > List of things to simulate(while heavily loaded), roughly in order of how > badly I think they need to be tested at the moment: > -Rpc connection disconnect > -Rpc connection slowed down enough to cause a timeout and trigger retry > -DN connection disconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10231) libhdfs++: Fix race conditions in RPC layer
[ https://issues.apache.org/jira/browse/HDFS-10231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10231: --- Attachment: HDFS-10231.HDFS-8707.002.patch New patch, now tests should pass. > libhdfs++: Fix race conditions in RPC layer > --- > > Key: HDFS-10231 > URL: https://issues.apache.org/jira/browse/HDFS-10231 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Bob Hansen > Attachments: HDFS-10231.HDFS-8707.000.patch, > HDFS-10231.HDFS-8707.001.patch, HDFS-10231.HDFS-8707.002.patch > > > Namenode calls seem to hang and the retry logic never properly kicks in. It > looks like there's a race condition that leads to a failed rpc call never > properly passing the request object to the new RpcConnectionImpl so things > hang forever. RpcConnectionImpl objects are also kept alive longer than they > should because of a shared_ptr cycle between them and the timeout tracking > objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10222) libhdfs++: Shutdown sockets to avoid "Connection reset by peer"
[ https://issues.apache.org/jira/browse/HDFS-10222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10222: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to HDFS-8707 > libhdfs++: Shutdown sockets to avoid "Connection reset by peer" > --- > > Key: HDFS-10222 > URL: https://issues.apache.org/jira/browse/HDFS-10222 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10222.HDFS-8707.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9908) Datanode should tolerate disk scan failure during NN handshake
[
https://issues.apache.org/jira/browse/HDFS-9908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-9908:
--
Attachment: HDFS-9908.007.patch
Post the patch again. For some reason precommit validation was not triggered.
v06 and v07 are the same.
> Datanode should tolerate disk scan failure during NN handshake
> --
>
> Key: HDFS-9908
> URL: https://issues.apache.org/jira/browse/HDFS-9908
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
> Environment: CDH5.3.3
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HDFS-9908.001.patch, HDFS-9908.002.patch,
> HDFS-9908.003.patch, HDFS-9908.004.patch, HDFS-9908.005.patch,
> HDFS-9908.006.patch, HDFS-9908.007.patch
>
>
> DN may treat a disk scan failure exception as an NN handshake exception, and
> this can prevent a DN to join a cluster even if most of its disks are healthy.
> During NN handshake, DN initializes block pools. It will create a lock files
> per disk, and then scan the volumes. However, if the scanning throws
> exceptions due to disk failure, DN will think it's an exception because NN is
> inconsistent with the local storage (see {{DataNode#initBlockPool}}. As a
> result, it will attempt to reconnect to NN again.
> However, at this point, DN has not deleted its lock files on the disks. If it
> reconnects to NN again, it will think the same disks are already being used,
> and then it will fail handshake again because all disks can not be used (due
> to locking), and repeatedly. This will happen even if the DN has multiple
> disks, and only one of them fails. The DN will not be able to connect to NN
> despite just one failing disk. Note that it is possible to successfully
> create a lock file on a disk, and then has error scanning the disk.
> We saw this on a CDH 5.3.3 cluster (which is based on Apache Hadoop 2.5.0,
> and we still see the same bug in 3.0.0 trunk branch). The root cause is that
> DN treats an internal error (single disk failure) as an external one (NN
> handshake failure) and we should fix it.
> {code:title=DataNode.java}
> /**
>* One of the Block Pools has successfully connected to its NN.
>* This initializes the local storage for that block pool,
>* checks consistency of the NN's cluster ID, etc.
>*
>* If this is the first block pool to register, this also initializes
>* the datanode-scoped storage.
>*
>* @param bpos Block pool offer service
>* @throws IOException if the NN is inconsistent with the local storage.
>*/
> void initBlockPool(BPOfferService bpos) throws IOException {
> NamespaceInfo nsInfo = bpos.getNamespaceInfo();
> if (nsInfo == null) {
> throw new IOException("NamespaceInfo not found: Block pool " + bpos
> + " should have retrieved namespace info before initBlockPool.");
> }
>
> setClusterId(nsInfo.clusterID, nsInfo.getBlockPoolID());
> // Register the new block pool with the BP manager.
> blockPoolManager.addBlockPool(bpos);
>
> // In the case that this is the first block pool to connect, initialize
> // the dataset, block scanners, etc.
> initStorage(nsInfo);
> // Exclude failed disks before initializing the block pools to avoid
> startup
> // failures.
> checkDiskError();
> data.addBlockPool(nsInfo.getBlockPoolID(), conf); <- this line
> throws disk error exception
> blockScanner.enableBlockPoolId(bpos.getBlockPoolId());
> initDirectoryScanner(conf);
> }
> {code}
> {{FsVolumeList#addBlockPool}} is the source of exception.
> {code:title=FsVolumeList.java}
> void addBlockPool(final String bpid, final Configuration conf) throws
> IOException {
> long totalStartTime = Time.monotonicNow();
>
> final List exceptions = Collections.synchronizedList(
> new ArrayList());
> List blockPoolAddingThreads = new ArrayList();
> for (final FsVolumeImpl v : volumes) {
> Thread t = new Thread() {
> public void run() {
> try (FsVolumeReference ref = v.obtainReference()) {
> FsDatasetImpl.LOG.info("Scanning block pool " + bpid +
> " on volume " + v + "...");
> long startTime = Time.monotonicNow();
> v.addBlockPool(bpid, conf);
> long timeTaken = Time.monotonicNow() - startTime;
> FsDatasetImpl.LOG.info("Time taken to scan block pool " + bpid +
> " on " + v + ": " + timeTaken + "ms");
> } catch (ClosedChannelException e) {
> // ignore.
> } catch (IOException ioe) {
> FsDatasetImpl.LOG.info("Caught exception while scanning " + v +
> ". Will throw later.", ioe);
>
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[
https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218541#comment-15218541
]
Chris Douglas commented on HDFS-9847:
-
bq. IMO, we should go ahead without changing the default values and breaking
tests in this Jira.
Previous versions of the patch were not correct. The unit tests caught this,
and trying to start a cluster using a config with units
[confirmed|https://issues.apache.org/jira/browse/HDFS-9847?focusedCommentId=15217465=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15217465]
it. The handful of tests that do not pass should not be difficult to fix.
Please file the JIRA before commit, so deprecation can be tracked.
> HDFS configuration without time unit name should accept friendly time units
> ---
>
> Key: HDFS-9847
> URL: https://issues.apache.org/jira/browse/HDFS-9847
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HDFS-9847-branch-2.001.patch,
> HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch,
> HDFS-9847-nothrow.002.patch, HDFS-9847-nothrow.003.patch,
> HDFS-9847.001.patch, HDFS-9847.002.patch, HDFS-9847.003.patch,
> HDFS-9847.004.patch, HDFS-9847.005.patch, HDFS-9847.006.patch,
> branch-2-delta.002.txt, timeduration-w-y.patch
>
>
> In HDFS-9821, it talks about the issue of leting existing keys use friendly
> units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names
> contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make
> some other configurations which without time unit name to accept friendly
> time units. The time unit {{seconds}} is frequently used in hdfs. We can
> updating this configurations first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9890) libhdfs++: Add test suite to simulate network issues
[ https://issues.apache.org/jira/browse/HDFS-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9890: -- Attachment: HDFS-9890.HDFS-8707.000.patch Initial patch, right now error injection is turned on and off with "#ifdef ENABLE_RPC_ERROR_SIMULATION". I think the hooks added in HDFS-9616 are the best way to trigger these conditions and give better control in more complicated tests. > libhdfs++: Add test suite to simulate network issues > > > Key: HDFS-9890 > URL: https://issues.apache.org/jira/browse/HDFS-9890 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Stephen > Attachments: HDFS-9890.HDFS-8707.000.patch > > > I propose adding a test suite to simulate various network issues/failures in > order to get good test coverage on some of the retry paths that aren't easy > to hit in mock unit tests. > At the moment the only things that hit the retry paths are the gmock unit > tests. The gmock are only as good as their mock implementations which do a > great job of simulating protocol correctness but not more complex > interactions. They also can't really simulate the types of lock contention > and subtle memory stomps that show up while doing hundreds or thousands of > concurrent reads. We should add a new minidfscluster test that focuses on > heavy read/seek load and then randomly convert error codes returned by > network functions into errors. > List of things to simulate(while heavily loaded), roughly in order of how > badly I think they need to be tested at the moment: > -Rpc connection disconnect > -Rpc connection slowed down enough to cause a timeout and trigger retry > -DN connection disconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-5280) Corrupted meta files on data nodes prevents DFClient from connecting to data nodes and updating corruption status to name node.
[ https://issues.apache.org/jira/browse/HDFS-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andres Perez updated HDFS-5280: --- Affects Version/s: 2.7.2 > Corrupted meta files on data nodes prevents DFClient from connecting to data > nodes and updating corruption status to name node. > --- > > Key: HDFS-5280 > URL: https://issues.apache.org/jira/browse/HDFS-5280 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs-client >Affects Versions: 1.1.1, 2.1.0-beta, 2.0.4-alpha, 2.7.2 > Environment: Red hat enterprise 6.4 > Hadoop-2.1.0 >Reporter: Jinghui Wang >Assignee: Andres Perez > > Meta files being corrupted causes the DFSClient not able to connect to the > datanodes to access the blocks, so DFSClient never perform a read on the > block, which is what throws the ChecksumException when file blocks are > corrupted and report to the namenode to mark the block as corrupt. Since the > client never got to that far, thus the file status remain as healthy and so > are all the blocks. > To replicate the error, put a file onto HDFS. > run hadoop fsck /tmp/bogus.csv -files -blocks -location will get that > following output. > FSCK started for path /tmp/bogus.csv at 11:33:29 > /tmp/bogus.csv 109 bytes, 1 block(s): OK > 0. blk_-4255166695856420554_5292 len=109 repl=3 > find the block/meta files for 4255166695856420554 by running > ssh datanode1.address find /hadoop/ -name "*4255166695856420554*" and it will > get the following output: > /hadoop/data1/hdfs/current/subdir2/blk_-4255166695856420554 > /hadoop/data1/hdfs/current/subdir2/blk_-4255166695856420554_5292.meta > now corrupt the meta file by running > ssh datanode1.address "sed -i -e '1i 1234567891' > /hadoop/data1/hdfs/current/subdir2/blk_-4255166695856420554_5292.meta" > now run hadoop fs -cat /tmp/bogus.csv > will show the stack trace of DFSClient failing to connect to the data node > with the corrupted meta file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9239) DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness
[ https://issues.apache.org/jira/browse/HDFS-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218470#comment-15218470 ] Nathan Roberts commented on HDFS-9239: -- bq. Just to make sure I'm clear, are you talking about configuring the deadline scheduler as described here? Yes, those links are talking about the right parameters. We currently run with read_expire=1000, write_expire=1000, and writes_starved=1. Since our I/O workloads change dramatically over time, we didn't spend a lot of time looking for optimal values here. These have been working well for the last several months across multiple clusters. As an aside, a relatively easy way to reproduce this problem, is to put a heavy seek load on all the disks of a datanode (e.g. http://www.linuxinsight.com/how_fast_is_your_disk.html, I believe 5-10 copies of seeker were sufficient.) After a minute or so, system becomes almost unusable and datanode will be declared lost. This might be a good test to run against the lifeline protocol. My hunch is, with CFQ, the datanode will still be lost. > DataNode Lifeline Protocol: an alternative protocol for reporting DataNode > liveness > --- > > Key: HDFS-9239 > URL: https://issues.apache.org/jira/browse/HDFS-9239 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 2.8.0 > > Attachments: DataNode-Lifeline-Protocol.pdf, HDFS-9239.001.patch, > HDFS-9239.002.patch, HDFS-9239.003.patch > > > This issue proposes introduction of a new feature: the DataNode Lifeline > Protocol. This is an RPC protocol that is responsible for reporting liveness > and basic health information about a DataNode to a NameNode. Compared to the > existing heartbeat messages, it is lightweight and not prone to resource > contention problems that can harm accurate tracking of DataNode liveness > currently. The attached design document contains more details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-10230) HDFS Native Client build failed
[
https://issues.apache.org/jira/browse/HDFS-10230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki resolved HDFS-10230.
-
Resolution: Duplicate
I'm going to close this as duplicate of HADOOP-12692. Please reopen if this
turns out to be independent issue.
> HDFS Native Client build failed
> ---
>
> Key: HDFS-10230
> URL: https://issues.apache.org/jira/browse/HDFS-10230
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Reporter: John Zhuge
>Priority: Blocker
>
> HDFS Native Client build failed:
> https://builds.apache.org/job/Hadoop-trunk-Commit/9514/console
> {code}
> [INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @
> hadoop-hdfs-native-client ---
> Downloading:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-hdfs/3.0.0-SNAPSHOT/hadoop-hdfs-3.0.0-20160328.214654-6500.pom
> 4/21 KB
> 8/21 KB
> 8/21 KB
> 12/21 KB
> 14/21 KB
> 18/21 KB
> 21/21 KB
>
> Downloaded:
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-hdfs/3.0.0-SNAPSHOT/hadoop-hdfs-3.0.0-20160328.214654-6500.pom
> (21 KB at 193.9 KB/sec)
> [WARNING]
> Dependency convergence error for org.apache.hadoop:hadoop-hdfs:3.0.0-SNAPSHOT
> paths to dependency are:
> +-org.apache.hadoop:hadoop-hdfs-native-client:3.0.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-hdfs:3.0.0-SNAPSHOT
> and
> +-org.apache.hadoop:hadoop-hdfs-native-client:3.0.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-hdfs:3.0.0-20160328.214654-6500
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence
> failed with message:
> Failed while enforcing releasability the error(s) are [
> Dependency convergence error for org.apache.hadoop:hadoop-hdfs:3.0.0-SNAPSHOT
> paths to dependency are:
> +-org.apache.hadoop:hadoop-hdfs-native-client:3.0.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-hdfs:3.0.0-SNAPSHOT
> and
> +-org.apache.hadoop:hadoop-hdfs-native-client:3.0.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-hdfs:3.0.0-20160328.214654-6500
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10231) libhdfs++: Fix race conditions in RPC layer
[ https://issues.apache.org/jira/browse/HDFS-10231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218397#comment-15218397 ] James Clampffer commented on HDFS-10231: Test failure is due to a bad merge I made; [~bobthansen] did the actual work for this patch. I'll get a fix up today. > libhdfs++: Fix race conditions in RPC layer > --- > > Key: HDFS-10231 > URL: https://issues.apache.org/jira/browse/HDFS-10231 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Bob Hansen > Attachments: HDFS-10231.HDFS-8707.000.patch, > HDFS-10231.HDFS-8707.001.patch > > > Namenode calls seem to hang and the retry logic never properly kicks in. It > looks like there's a race condition that leads to a failed rpc call never > properly passing the request object to the new RpcConnectionImpl so things > hang forever. RpcConnectionImpl objects are also kept alive longer than they > should because of a shared_ptr cycle between them and the timeout tracking > objects. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9239) DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness
[ https://issues.apache.org/jira/browse/HDFS-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218396#comment-15218396 ] Chris Nauroth commented on HDFS-9239: - [~nroberts], fascinating! Thank you for sharing. Just to make sure I'm clear, are you talking about configuring the deadline scheduler as described here? https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Tuning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/sect-Oracle_9i_and_10g_Tuning_Guide-Kernel_Boot_Parameters-The_IO_Scheduler.html Also, did you find any relevant additional scheduler tuning configuration was needed, as described here? https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/ch06s04s02.html > DataNode Lifeline Protocol: an alternative protocol for reporting DataNode > liveness > --- > > Key: HDFS-9239 > URL: https://issues.apache.org/jira/browse/HDFS-9239 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 2.8.0 > > Attachments: DataNode-Lifeline-Protocol.pdf, HDFS-9239.001.patch, > HDFS-9239.002.patch, HDFS-9239.003.patch > > > This issue proposes introduction of a new feature: the DataNode Lifeline > Protocol. This is an RPC protocol that is responsible for reporting liveness > and basic health information about a DataNode to a NameNode. Compared to the > existing heartbeat messages, it is lightweight and not prone to resource > contention problems that can harm accurate tracking of DataNode liveness > currently. The attached design document contains more details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10237) Support specifying checksum type in WebHDFS/HTTPFS writers
[
https://issues.apache.org/jira/browse/HDFS-10237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218387#comment-15218387
]
Hadoop QA commented on HDFS-10237:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 1s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 39s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 33s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 47s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 34s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 31s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 56s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 49s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 12s
{color} | {color:red} root: patch generated 18 new + 758 unchanged - 10 fixed =
776 total (was 768) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 54s
{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-hdfs-httpfs in the patch failed with JDK
v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 7m 18s
{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-httpfs-jdk1.7.0_95 with
JDK v1.7.0_95 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 25s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 52s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 7s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 15s
{color} | {color:green} hadoop-hdfs-httpfs in the patch
[jira] [Commented] (HDFS-9239) DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness
[ https://issues.apache.org/jira/browse/HDFS-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218302#comment-15218302 ] Nathan Roberts commented on HDFS-9239: -- bq. However,making it lighter on the datanode side is a good idea. We have seen many cases where nodes are declared dead because the service actor thread is delayed/blocked. Just a quick update on this comment. Even after HDFS-7060 we still had cases where Datanodes would fail to heartbeat in. We eventually tracked this down to the RHEL CFQ I/O scheduler. There are situations where significant seek activity (like a massive shuffle) can cause this I/O scheduler to indefinitely starve writers. This eventually causes the datanode and/or nodemanager processes to completely stop (probably due to logging I/O backing up). So, no matter how smart we make these daemons, they are going to be lost from the NN/RM point of view in these situations. But, this is actually probably the right thing to do in these cases, these daemons are clearly not able to do their job so SHOULD be declared lost. In any event, the change which we found most valuable for this situation was to use the deadline I/O scheduler. This dramatically improved the number of lost datanodes and nodemanagers we were seeing. > DataNode Lifeline Protocol: an alternative protocol for reporting DataNode > liveness > --- > > Key: HDFS-9239 > URL: https://issues.apache.org/jira/browse/HDFS-9239 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 2.8.0 > > Attachments: DataNode-Lifeline-Protocol.pdf, HDFS-9239.001.patch, > HDFS-9239.002.patch, HDFS-9239.003.patch > > > This issue proposes introduction of a new feature: the DataNode Lifeline > Protocol. This is an RPC protocol that is responsible for reporting liveness > and basic health information about a DataNode to a NameNode. Compared to the > existing heartbeat messages, it is lightweight and not prone to resource > contention problems that can harm accurate tracking of DataNode liveness > currently. The attached design document contains more details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9239) DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness
[ https://issues.apache.org/jira/browse/HDFS-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218270#comment-15218270 ] Chris Nauroth commented on HDFS-9239: - I also am hesitant to backport a patch of this size right now, but I might reconsider after getting more production experience with the feature. > DataNode Lifeline Protocol: an alternative protocol for reporting DataNode > liveness > --- > > Key: HDFS-9239 > URL: https://issues.apache.org/jira/browse/HDFS-9239 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 2.8.0 > > Attachments: DataNode-Lifeline-Protocol.pdf, HDFS-9239.001.patch, > HDFS-9239.002.patch, HDFS-9239.003.patch > > > This issue proposes introduction of a new feature: the DataNode Lifeline > Protocol. This is an RPC protocol that is responsible for reporting liveness > and basic health information about a DataNode to a NameNode. Compared to the > existing heartbeat messages, it is lightweight and not prone to resource > contention problems that can harm accurate tracking of DataNode liveness > currently. The attached design document contains more details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10220) Namenode failover due to too long loking in LeaseManager.Monitor
[ https://issues.apache.org/jira/browse/HDFS-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218247#comment-15218247 ] Nicolas Fraison commented on HDFS-10220: [~vinayrpet] the day we face this kind of failover we have faced multiple failover with the same issue on both namenodes. It happens after a bad action on the cluster removing the mapreduce.jobhistory.intermediate-done-dir folder whith then lots of mapreduce failing... Since we have applied this patch we have one time reached 250K lease to release taking a total time of 45 seconds (having 100 k lease treated per cycle). > Namenode failover due to too long loking in LeaseManager.Monitor > > > Key: HDFS-10220 > URL: https://issues.apache.org/jira/browse/HDFS-10220 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Nicolas Fraison >Priority: Minor > Attachments: HADOOP-10220.001.patch, threaddump_zkfc.txt > > > I have faced a namenode failover due to unresponsive namenode detected by the > zkfc with lot's of WARN messages (5 millions) like this one: > _org.apache.hadoop.hdfs.StateChange: BLOCK* internalReleaseLease: All > existing blocks are COMPLETE, lease removed, file closed._ > On the threaddump taken by the zkfc there are lots of thread blocked due to a > lock. > Looking at the code, there are a lock taken by the LeaseManager.Monitor when > some lease must be released. Due to the really big number of lease to be > released the namenode has taken too many times to release them blocking all > other tasks and making the zkfc thinking that the namenode was not > available/stuck. > The idea of this patch is to limit the number of leased released each time we > check for lease so the lock won't be taken for a too long time period. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10222) libhdfs++: Shutdown sockets to avoid "Connection reset by peer"
[ https://issues.apache.org/jira/browse/HDFS-10222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218235#comment-15218235 ] James Clampffer commented on HDFS-10222: Thanks for reviewing Stephen! As long as the users follow the 1 thread : 1 file handle (except for relaxed position read rules) cancel doesn't need to be thread safe. > libhdfs++: Shutdown sockets to avoid "Connection reset by peer" > --- > > Key: HDFS-10222 > URL: https://issues.apache.org/jira/browse/HDFS-10222 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10222.HDFS-8707.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9869) Erasure Coding: Rename replication-based names in BlockManager to more generic [part-2]
[
https://issues.apache.org/jira/browse/HDFS-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-9869:
---
Attachment: HDFS-9869-003.patch
Attached patch fixing patch related {{deprecated}} javac warnings and
checkstyle warnings.
> Erasure Coding: Rename replication-based names in BlockManager to more
> generic [part-2]
> ---
>
> Key: HDFS-9869
> URL: https://issues.apache.org/jira/browse/HDFS-9869
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-9869-001.patch, HDFS-9869-002.patch,
> HDFS-9869-003.patch
>
>
> The idea of this jira is to rename the following entities in BlockManager as,
> - {{PendingReplicationBlocks}} to {{PendingReconstructionBlocks}}
> - {{excessReplicateMap}} to {{extraRedundancyMap}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10222) libhdfs++: Shutdown sockets to avoid "Connection reset by peer"
[ https://issues.apache.org/jira/browse/HDFS-10222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218171#comment-15218171 ] Stephen commented on HDFS-10222: +1, provided calls to Cancel don't need to be thread safe. > libhdfs++: Shutdown sockets to avoid "Connection reset by peer" > --- > > Key: HDFS-10222 > URL: https://issues.apache.org/jira/browse/HDFS-10222 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10222.HDFS-8707.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10220) Namenode failover due to too long loking in LeaseManager.Monitor
[
https://issues.apache.org/jira/browse/HDFS-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218106#comment-15218106
]
Ravi Prakash commented on HDFS-10220:
-
Also lets rename {{maxPathReleaseExpiredLease}} to
{{maxFilesCheckedForRelease}} (because {{completed}} may be false in some cases.
> Namenode failover due to too long loking in LeaseManager.Monitor
>
>
> Key: HDFS-10220
> URL: https://issues.apache.org/jira/browse/HDFS-10220
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Nicolas Fraison
>Priority: Minor
> Attachments: HADOOP-10220.001.patch, threaddump_zkfc.txt
>
>
> I have faced a namenode failover due to unresponsive namenode detected by the
> zkfc with lot's of WARN messages (5 millions) like this one:
> _org.apache.hadoop.hdfs.StateChange: BLOCK* internalReleaseLease: All
> existing blocks are COMPLETE, lease removed, file closed._
> On the threaddump taken by the zkfc there are lots of thread blocked due to a
> lock.
> Looking at the code, there are a lock taken by the LeaseManager.Monitor when
> some lease must be released. Due to the really big number of lease to be
> released the namenode has taken too many times to release them blocking all
> other tasks and making the zkfc thinking that the namenode was not
> available/stuck.
> The idea of this patch is to limit the number of leased released each time we
> check for lease so the lock won't be taken for a too long time period.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10220) Namenode failover due to too long loking in LeaseManager.Monitor
[
https://issues.apache.org/jira/browse/HDFS-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218097#comment-15218097
]
Ravi Prakash commented on HDFS-10220:
-
{code}import static org.apache.hadoop.hdfs.DFSConfigKeys.*;{code}
Could you please import classes explicitly?
{code}
/** Number max of path for released lease each time Monitor check for expired
lease */
private final long maxPathRealeaseExpiredLease;
{code}
has grammar and spelling errors. I'd suggest
{code}
/** Maximum number of files whose lease will be released in one iteration of
checkLeases() */
private final long maxPathReleaseExpiredLease; // <-- Release was misspelt
here
{code}
{code}
Configuration conf = new Configuration();
this.maxPathRealeaseExpiredLease =
conf.getLong(DFS_NAMENODE_MAX_PATH_RELEASE_EXPIRED_LEASE_KEY,
DFS_NAMENODE_MAX_PATH_RELEASE_EXPIRED_LEASE_DEFAULT);
{code}
I'm fine with not getting {{maxPathRealeaseExpiredLease}} from configuration
and hardcoding it to your default value of 10. If you want to keep the
configuration, I'd suggest changing
{{dfs.namenode.max-path-release-expired-lease}} to
{{dfs.namenode.lease-manager.max-released-leases-per-iteration}} .
Please rename {{nPathReleaseExpiredLease}} to {{numLeasesReleased}}
{code}// Stop releasing lease as a lock is hold after few iterations{code}
change to {code} //Relinquish FSNamesystem lock after
maxPathRealeaseExpiredLease iterations {code}
{code}LOG.warn("Breaking out of checkLeases() after " +
nPathReleaseExpiredLease + " iterations",
new Throwable("Too long loop with a lock"));
{code}
Its unnecessary to log an exception.
For the purposes of test, you can add a method which changes the
maxPathRealeaseExpiredLease and annotate it Could you please also also rename
{{testCheckLeaseNotInfiniteLoop}} and change its documentation.
> Namenode failover due to too long loking in LeaseManager.Monitor
>
>
> Key: HDFS-10220
> URL: https://issues.apache.org/jira/browse/HDFS-10220
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Nicolas Fraison
>Priority: Minor
> Attachments: HADOOP-10220.001.patch, threaddump_zkfc.txt
>
>
> I have faced a namenode failover due to unresponsive namenode detected by the
> zkfc with lot's of WARN messages (5 millions) like this one:
> _org.apache.hadoop.hdfs.StateChange: BLOCK* internalReleaseLease: All
> existing blocks are COMPLETE, lease removed, file closed._
> On the threaddump taken by the zkfc there are lots of thread blocked due to a
> lock.
> Looking at the code, there are a lock taken by the LeaseManager.Monitor when
> some lease must be released. Due to the really big number of lease to be
> released the namenode has taken too many times to release them blocking all
> other tasks and making the zkfc thinking that the namenode was not
> available/stuck.
> The idea of this patch is to limit the number of leased released each time we
> check for lease so the lock won't be taken for a too long time period.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10220) Namenode failover due to too long loking in LeaseManager.Monitor
[ https://issues.apache.org/jira/browse/HDFS-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218036#comment-15218036 ] Ravi Prakash commented on HDFS-10220: - Oh! I'm sorry in that case. Thank you for re-opening the JIRA. Just out of curiosity, how many files did you have open at one time? Do you think we should cycle through all the leases rather than the same set every iteration? We may be over-engineering, but I'd be interested in your opinion. I'll review your patch shortly > Namenode failover due to too long loking in LeaseManager.Monitor > > > Key: HDFS-10220 > URL: https://issues.apache.org/jira/browse/HDFS-10220 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Nicolas Fraison >Priority: Minor > Attachments: HADOOP-10220.001.patch, threaddump_zkfc.txt > > > I have faced a namenode failover due to unresponsive namenode detected by the > zkfc with lot's of WARN messages (5 millions) like this one: > _org.apache.hadoop.hdfs.StateChange: BLOCK* internalReleaseLease: All > existing blocks are COMPLETE, lease removed, file closed._ > On the threaddump taken by the zkfc there are lots of thread blocked due to a > lock. > Looking at the code, there are a lock taken by the LeaseManager.Monitor when > some lease must be released. Due to the really big number of lease to be > released the namenode has taken too many times to release them blocking all > other tasks and making the zkfc thinking that the namenode was not > available/stuck. > The idea of this patch is to limit the number of leased released each time we > check for lease so the lock won't be taken for a too long time period. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-6542) WebHDFSFileSystem doesn't transmit desired checksum type
[ https://issues.apache.org/jira/browse/HDFS-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HDFS-6542. --- Resolution: Duplicate I missed this JIRA when searching before I filed HDFS-10237, but now noticed via association to HADOOP-8240. Since I've already posted a patch on HDFS-10237 and there's no ongoing work/assignee here, am marking this as a duplicate of HDFS-10237. Sorry for the extra noise! > WebHDFSFileSystem doesn't transmit desired checksum type > > > Key: HDFS-6542 > URL: https://issues.apache.org/jira/browse/HDFS-6542 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Andrey Stepachev >Priority: Minor > > Currently DFSClient has possibility to specify desired checksum type. This > behaviour controlled by dfs.checksym.type parameter settable by client. > It works with hdfs:// filesystem, but doesn't works with webhdfs.It fails to > work because webhdfs will use default type of checksumming initialised by > server instance of DFSClient. > As example https://issues.apache.org/jira/browse/HADOOP-8240 doesn't works > with webhdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10237) Support specifying checksum type in WebHDFS/HTTPFS writers
[ https://issues.apache.org/jira/browse/HDFS-10237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-10237: --- Attachment: HDFS-10237.000.patch > Support specifying checksum type in WebHDFS/HTTPFS writers > -- > > Key: HDFS-10237 > URL: https://issues.apache.org/jira/browse/HDFS-10237 > Project: Hadoop HDFS > Issue Type: New Feature > Components: webhdfs >Affects Versions: 2.8.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Minor > Attachments: HDFS-10237.000.patch > > > Currently you cannot set a desired checksum type over a WebHDFS or HTTPFS > writer, as you can with the regular DFS writer (done via HADOOP-8240) > This JIRA covers the changes necessary to bring the same ability to WebHDFS > and HTTPFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10237) Support specifying checksum type in WebHDFS/HTTPFS writers
[ https://issues.apache.org/jira/browse/HDFS-10237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-10237: --- Status: Patch Available (was: Open) > Support specifying checksum type in WebHDFS/HTTPFS writers > -- > > Key: HDFS-10237 > URL: https://issues.apache.org/jira/browse/HDFS-10237 > Project: Hadoop HDFS > Issue Type: New Feature > Components: webhdfs >Affects Versions: 2.8.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Minor > Attachments: HDFS-10237.000.patch > > > Currently you cannot set a desired checksum type over a WebHDFS or HTTPFS > writer, as you can with the regular DFS writer (done via HADOOP-8240) > This JIRA covers the changes necessary to bring the same ability to WebHDFS > and HTTPFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-10237) Support specifying checksum type in WebHDFS/HTTPFS writers
Harsh J created HDFS-10237: -- Summary: Support specifying checksum type in WebHDFS/HTTPFS writers Key: HDFS-10237 URL: https://issues.apache.org/jira/browse/HDFS-10237 Project: Hadoop HDFS Issue Type: New Feature Components: webhdfs Affects Versions: 2.8.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Currently you cannot set a desired checksum type over a WebHDFS or HTTPFS writer, as you can with the regular DFS writer (done via HADOOP-8240) This JIRA covers the changes necessary to bring the same ability to WebHDFS and HTTPFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9869) Erasure Coding: Rename replication-based names in BlockManager to more generic [part-2]
[
https://issues.apache.org/jira/browse/HDFS-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217930#comment-15217930
]
Hadoop QA commented on HDFS-9869:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 10 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 34s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 45s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 31s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 45s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 56s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 48s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 13s {color}
| {color:red} root-jdk1.8.0_77 with JDK v1.8.0_77 generated 23 new + 737
unchanged - 2 fixed = 760 total (was 739) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 44s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 57s
{color} | {color:red} root-jdk1.7.0_95 with JDK v1.7.0_95 generated 23 new +
733 unchanged - 2 fixed = 756 total (was 735) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 44s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 13s
{color} | {color:red} root: patch generated 4 new + 748 unchanged - 3 fixed =
752 total (was 751) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 25s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 17s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 22s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} |
[jira] [Assigned] (HDFS-8065) Erasure coding: Support truncate at striped group boundary.
[ https://issues.apache.org/jira/browse/HDFS-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R reassigned HDFS-8065: -- Assignee: Rakesh R (was: Yi Liu) > Erasure coding: Support truncate at striped group boundary. > --- > > Key: HDFS-8065 > URL: https://issues.apache.org/jira/browse/HDFS-8065 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Yi Liu >Assignee: Rakesh R > > We can support truncate at striped group boundary firstly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-10236) Erasure Coding: Rename replication-based names in BlockManager to more generic [part-3]
Rakesh R created HDFS-10236:
---
Summary: Erasure Coding: Rename replication-based names in
BlockManager to more generic [part-3]
Key: HDFS-10236
URL: https://issues.apache.org/jira/browse/HDFS-10236
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
The idea of this jira is to rename the following entity in BlockManager as,
{{getExpectedReplicaNum}} to {{getExpectedRedundancyNum}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9918) Erasure Coding: Sort located striped blocks based on decommissioned states
[
https://issues.apache.org/jira/browse/HDFS-9918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217773#comment-15217773
]
Hadoop QA commented on HDFS-9918:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 44s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 42s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 9s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 20s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 167m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests | hadoop.hdfs.TestHFlush |
| | hadoop.hdfs.TestFileAppend |
| JDK v1.8.0_74 Timed out junit tests |
org.apache.hadoop.hdfs.TestLeaseRecovery2 |
| JDK v1.7.0_95 Failed junit tests | hadoop.hdfs.TestHFlush |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12796030/HDFS-9918-008.patch |
| JIRA Issue | HDFS-9918 |
| Optional Tests | asflicense compile javac javadoc mvninstall
[jira] [Commented] (HDFS-9869) Erasure Coding: Rename replication-based names in BlockManager to more generic [part-2]
[
https://issues.apache.org/jira/browse/HDFS-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217746#comment-15217746
]
Rakesh R commented on HDFS-9869:
Thanks [~zhz], [~andrew.wang] for the advice and comments.
Attached another patch addressing the comments.
> Erasure Coding: Rename replication-based names in BlockManager to more
> generic [part-2]
> ---
>
> Key: HDFS-9869
> URL: https://issues.apache.org/jira/browse/HDFS-9869
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-9869-001.patch, HDFS-9869-002.patch
>
>
> The idea of this jira is to rename the following entities in BlockManager as,
> - {{PendingReplicationBlocks}} to {{PendingReconstructionBlocks}}
> - {{excessReplicateMap}} to {{extraRedundancyMap}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9869) Erasure Coding: Rename replication-based names in BlockManager to more generic [part-2]
[
https://issues.apache.org/jira/browse/HDFS-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-9869:
---
Attachment: HDFS-9869-002.patch
> Erasure Coding: Rename replication-based names in BlockManager to more
> generic [part-2]
> ---
>
> Key: HDFS-9869
> URL: https://issues.apache.org/jira/browse/HDFS-9869
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-9869-001.patch, HDFS-9869-002.patch
>
>
> The idea of this jira is to rename the following entities in BlockManager as,
> - {{PendingReplicationBlocks}} to {{PendingReconstructionBlocks}}
> - {{excessReplicateMap}} to {{extraRedundancyMap}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[
https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217728#comment-15217728
]
Hadoop QA commented on HDFS-9847:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 49s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 49s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 45s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 17s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 20s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 15s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 56s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s {color} | {color:green} root: patch generated 0 new + 1002 unchanged - 2
fixed = 1002 total (was 1004) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 20s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 11s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 34s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 53s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 56s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 57s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 59s
{color} | {color:green} hadoop-hdfs-client in the patch passed with
[jira] [Commented] (HDFS-9239) DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness
[ https://issues.apache.org/jira/browse/HDFS-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217719#comment-15217719 ] Steve Loughran commented on HDFS-9239: -- I'm nervous about pushing stuff back. At the very least, ship Hadoop 2.8 and see if things break > DataNode Lifeline Protocol: an alternative protocol for reporting DataNode > liveness > --- > > Key: HDFS-9239 > URL: https://issues.apache.org/jira/browse/HDFS-9239 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 2.8.0 > > Attachments: DataNode-Lifeline-Protocol.pdf, HDFS-9239.001.patch, > HDFS-9239.002.patch, HDFS-9239.003.patch > > > This issue proposes introduction of a new feature: the DataNode Lifeline > Protocol. This is an RPC protocol that is responsible for reporting liveness > and basic health information about a DataNode to a NameNode. Compared to the > existing heartbeat messages, it is lightweight and not prone to resource > contention problems that can harm accurate tracking of DataNode liveness > currently. The attached design document contains more details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217716#comment-15217716
]
Steve Loughran commented on HDFS-9263:
--
Not looked at this for a while; been busy with non-hadoop stuff. You want to
take over?
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10221) Add test resource dfs.hosts.json to the rat exclusions
[ https://issues.apache.org/jira/browse/HDFS-10221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217712#comment-15217712 ] Steve Loughran commented on HDFS-10221: --- Licenses are happy +1 > Add test resource dfs.hosts.json to the rat exclusions > -- > > Key: HDFS-10221 > URL: https://issues.apache.org/jira/browse/HDFS-10221 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 2.8.0 > Environment: Jenkins >Reporter: Ming Ma >Assignee: Ming Ma >Priority: Blocker > Attachments: HDFS-10221-2.patch, HDFS-10221.patch > > > A new test resource dfs.hosts.json was added to HDFS-9005 for better > readability. Given json file doesn't allow comments, it violates ASF license. > To address this, we can add the file to rat exclusions list in the pom.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9239) DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness
[ https://issues.apache.org/jira/browse/HDFS-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217710#comment-15217710 ] Vinayakumar B commented on HDFS-9239: - Even though this Jira along with HDFS-9311 is New Feature, IMO it would be worth to merge to Branch-2.7. Lot of deployments would be benefited. What you think [~kihwal]? If okay, I would be happy to post a patch for branch-2.7, if available patches not apply on branch-2.7. > DataNode Lifeline Protocol: an alternative protocol for reporting DataNode > liveness > --- > > Key: HDFS-9239 > URL: https://issues.apache.org/jira/browse/HDFS-9239 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 2.8.0 > > Attachments: DataNode-Lifeline-Protocol.pdf, HDFS-9239.001.patch, > HDFS-9239.002.patch, HDFS-9239.003.patch > > > This issue proposes introduction of a new feature: the DataNode Lifeline > Protocol. This is an RPC protocol that is responsible for reporting liveness > and basic health information about a DataNode to a NameNode. Compared to the > existing heartbeat messages, it is lightweight and not prone to resource > contention problems that can harm accurate tracking of DataNode liveness > currently. The attached design document contains more details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217677#comment-15217677
]
Akira AJISAKA commented on HDFS-9263:
-
Hi [~ste...@apache.org], how is the issue going?
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-10178) Permanent write failures can happen if pipeline recoveries occur for the first packet
[
https://issues.apache.org/jira/browse/HDFS-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217655#comment-15217655
]
Akira AJISAKA commented on HDFS-10178:
--
Mostly looks good to me. Minor nit:
{code}
cluster = new MiniDFSCluster.Builder(conf).numDataNodes((int)3).build();
{code}
{{(int)}} is unnecessarily.
Hi [~cnauroth] and [~arpitagarwal], would you please review the patch?
> Permanent write failures can happen if pipeline recoveries occur for the
> first packet
> -
>
> Key: HDFS-10178
> URL: https://issues.apache.org/jira/browse/HDFS-10178
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Critical
> Attachments: HDFS-10178.patch, HDFS-10178.v2.patch,
> HDFS-10178.v3.patch
>
>
> We have observed that write fails permanently if the first packet doesn't go
> through properly and pipeline recovery happens. If the packet header is sent
> out, but the data portion of the packet does not reach one or more datanodes
> in time, the pipeline recovery will be done against the 0-byte partial block.
>
> If additional datanodes are added, the block is transferred to the new nodes.
> After the transfer, each node will have a meta file containing the header
> and 0-length data block file. The pipeline recovery seems to work correctly
> up to this point, but write fails when actual data packet is resent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9481) Expose reservedForReplicas as a metric
[ https://issues.apache.org/jira/browse/HDFS-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217654#comment-15217654 ] Brahma Reddy Battula commented on HDFS-9481: [~arpitagarwal] Seems to be duplicate of HDFS-6955..? It's exposed in HDFS-6955. > Expose reservedForReplicas as a metric > --- > > Key: HDFS-9481 > URL: https://issues.apache.org/jira/browse/HDFS-9481 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10208) Addendum for HDFS-9579: to handle the case when client machine can't resolve network path
[
https://issues.apache.org/jira/browse/HDFS-10208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217642#comment-15217642
]
Brahma Reddy Battula commented on HDFS-10208:
-
Thanks for taking care my
[comment|https://issues.apache.org/jira/browse/HDFS-9579?focusedCommentId=15212860=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15212860]..From
now onwards user needs to configure Mapping impl ( and its related values such
as script for ScriptBasedMapping) when the client-side network path resolution
is enabled. can you mention same in property description.
> Addendum for HDFS-9579: to handle the case when client machine can't resolve
> network path
> -
>
> Key: HDFS-10208
> URL: https://issues.apache.org/jira/browse/HDFS-10208
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HDFS-10208-2.patch, HDFS-10208.patch
>
>
> If DFSClient runs on a machine that can't resolve network path,
> e.g.{{dnsToSwitchMapping.resolve}} returns null, that will cause exception
> when it tries to create {{clientNode}}. In such case, there is no need to
> create {{clientNode}} as null {{clientNode}} means its network distance with
> any datanode is Integer.MAX_VALUE, which is what we want.
> {noformat}
> clientNode = new NodeBase(clientHostName,
> dnsToSwitchMapping.resolve(nodes).get(0));
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8976) Create HTML5 cluster webconsole for federated cluster
[
https://issues.apache.org/jira/browse/HDFS-8976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217638#comment-15217638
]
Hadoop QA commented on HDFS-8976:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 17s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 8m 45s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12755403/HDFS-8976-03.patch |
| JIRA Issue | HDFS-8976 |
| Optional Tests | asflicense mvnsite |
| uname | Linux 3e17f5e895f4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 09d63d5 |
| asflicense |
https://builds.apache.org/job/PreCommit-HDFS-Build/15000/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15000/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Create HTML5 cluster webconsole for federated cluster
> -
>
> Key: HDFS-8976
> URL: https://issues.apache.org/jira/browse/HDFS-8976
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 2.7.0
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-8976-01.patch, HDFS-8976-02.patch,
> HDFS-8976-03.patch, cluster-health.JPG
>
>
> Since the old jsp variant of cluster web console is no longer present from
> 2.7 onwards, there is a need for HTML 5 web console for overview of overall
> cluster.
> 2.7.1 docs says to check webconsole as below {noformat}Similar to the
> Namenode status web page, when using federation a Cluster Web Console is
> available to monitor the federated cluster at
> http:///dfsclusterhealth.jsp. Any Namenode in the cluster
> can be used to access this web page.{noformat}
> But this is no longer present,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
