[jira] [Commented] (HDFS-11196) Ozone: Improve logging and error handling in the container layer
[
https://issues.apache.org/jira/browse/HDFS-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893871#comment-15893871
]
Hadoop QA commented on HDFS-11196:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
38s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 98 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 0
unchanged - 6 fixed = 0 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
12s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
2s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}119m 22s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}152m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Redundant nullcheck of db, which is known to be non-null in
org.apache.hadoop.ozone.container.common.impl.KeyManagerImpl.deleteKey(Pipeline,
String) Redundant null check at KeyManagerImpl.java:is known to be non-null
in
org.apache.hadoop.ozone.container.common.impl.KeyManagerImpl.deleteKey(Pipeline,
String) Redundant null check at KeyManagerImpl.java:[line 155] |
| | Redundant nullcheck of db, which is known to be non-null in
org.apache.hadoop.ozone.container.common.impl.KeyManagerImpl.getKey(KeyData)
Redundant null check at KeyManagerImpl.java:is known to be non-null in
org.apache.hadoop.ozone.container.common.impl.KeyManagerImpl.getKey(KeyData)
Redundant null check at KeyManagerImpl.java:[line 118] |
| | Redundant nullcheck of db, which is known to be
[jira] [Commented] (HDFS-8741) Proper error msg to be printed when invalid operation type is given to WebHDFS operations.
[
https://issues.apache.org/jira/browse/HDFS-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893866#comment-15893866
]
Hadoop QA commented on HDFS-8741:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
56s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-8741 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855777/HDFS-8741.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e9442bbfc2a8 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 3749152 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18533/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18533/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Comment Edited] (HDFS-8409) HDFS client RPC call throws "java.lang.IllegalStateException"
[
https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892062#comment-15892062
]
Yongtao Yang edited comment on HDFS-8409 at 3/3/17 7:12 AM:
I found a clue that may be useful to this problem. {{DFSClient#namenode}} is a
proxy whose {{InvocationHandler}} is {{RetryInvocationHandler}}. If
{{namenode.toString()}} or {{namenode.hashCode()}} or some other non-rpc
methods are called, then the next real rpc method will failed at
{{setCallIdAndRetryCount()}}. The cause of this is that {{toString()}} is also
forwarded to {{RetryInvocationHandler.invoke()}} and
{{setCallIdAndRetryCount()}} is executed there, then
{{ipc.Client#callId.get()}} will not be {{null}} any more. For real rpc
methods, {{ipc.Client#callId}} will be reset to the value {{null}} when
creating a {{Call}} instance({{org.apache.hadoop.ipc.Client.Call.Call(RpcKind,
Writable)}}), but for {{toString()}} or {{hashCode()}}, it can't reach the
{{Call}} constructor, so {{ipc.Client#callId}} will never recover to be
{{null}} and the next method(whether it is a real rpc or not) will fail.
was (Author: ytyang):
I found a clue that may be useful to this problem. {{DFSClient#namenode}} is a
proxy whose {{InvocationHandler}} is {{RetryInvocationHandler}}. If
{{namenode.toString()}} or {{namenode.hashCode()}} or some other non-rpc
methods are called, then the next real rpc method will failed at
{{setCallIdAndRetryCount()}}. The cause of this is that {{toString()}} is also
forwarded to {{RetryInvocationHandler.invoke()}} and
{{setCallIdAndRetryCount()}} is executed there, then
{{ipc.Client#callId.get()}} will not be null any more. For real rpc methods,
{{ipc.Client#callId}} will be reset to null when creating a {{Call}}
instance({{org.apache.hadoop.ipc.Client.Call.Call(RpcKind, Writable)}}), but
for {{toString()}} or {{hashCode()}}, it can't reach the {{Call}} constructor,
so {{ipc.Client#callId}} will never recover to be null and the next
method(whether it is a real rpc or not) will fail.
> HDFS client RPC call throws "java.lang.IllegalStateException"
> -
>
> Key: HDFS-8409
> URL: https://issues.apache.org/jira/browse/HDFS-8409
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Juan Yu
>Assignee: Juan Yu
> Attachments: HDFS-8409.001.patch, HDFS-8409.002.patch,
> HDFS-8409.003.patch
>
>
> When the HDFS client RPC calls need to retry, it sometimes throws
> "java.lang.IllegalStateException" and retry is aborted and cause the client
> call will fail.
> {code}
> Caused by: java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99)
> at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
> {code}
> Here is the check that throws exception
> {code}
> public static void setCallIdAndRetryCount(int cid, int rc) {
> ...
> Preconditions.checkState(callId.get() == null);
> }
> {code}
> The RetryInvocationHandler tries to call it with not null callId and causes
> exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8741) Proper error msg to be printed when invalid operation type is given to WebHDFS operations.
[
https://issues.apache.org/jira/browse/HDFS-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893787#comment-15893787
]
Yiqun Lin commented on HDFS-8741:
-
Would you add message into {{fail()}} as well? I see it still empty message
there.
{code}
+ public void testHttpOpParams() {
+try {
+ new PostOpParam("TEST");
+ Assert.fail();<==
+} catch (IllegalArgumentException e) {
+ GenericTestUtils
+ .assertExceptionContains("TEST is not a valid POST operation.", e);
+}
{code}
> Proper error msg to be printed when invalid operation type is given to
> WebHDFS operations.
> --
>
> Key: HDFS-8741
> URL: https://issues.apache.org/jira/browse/HDFS-8741
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Attachments: HDFS-8741.001.patch, HDFS-8741.002.patch,
> HDFS-8741.003.patch, HDFS-8741.004.patch
>
>
> When wrong operation type is given to WebHDFS operations, following Error
> message is printed --
> For ex: CREATE is called with GET instead of PUT--
> HTTP/1.1 400 Bad Request
> ..
> {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Invalid
> value for webhdfs parameter \"op\": {color:red}No enum constant
> org.apache.hadoop.hdfs.web.resources.PutOpParam.Op.CREATE"}}{color}
> Expected--
> Valid Error message to be printed
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11493) Ozone: SCM: Add the ability to handle container reports
Anu Engineer created HDFS-11493: --- Summary: Ozone: SCM: Add the ability to handle container reports Key: HDFS-11493 URL: https://issues.apache.org/jira/browse/HDFS-11493 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Anu Engineer Assignee: Anu Engineer Once a datanode sends the container report it is SCM's responsibility to determine if the replication levels are acceptable. If it is not, SCM should initiate a replication request to another datanode. This JIRA tracks how SCM handles a container report. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11492) Ozone: Add the ability to handle sendContainerReport Command
Anu Engineer created HDFS-11492: --- Summary: Ozone: Add the ability to handle sendContainerReport Command Key: HDFS-11492 URL: https://issues.apache.org/jira/browse/HDFS-11492 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Anu Engineer Assignee: Anu Engineer Once a container report is ready datanodes sends that information to SCM. SCM returns a command asking the data node to send container report. Add the ability to handle this command on datanode side and send the actual container Report. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11491) Ozone: Add close container RPC
Anu Engineer created HDFS-11491: --- Summary: Ozone: Add close container RPC Key: HDFS-11491 URL: https://issues.apache.org/jira/browse/HDFS-11491 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: HDFS-7240 A container can be in open state or closed state. A closed container is immutable. This immutability makes it easy to operate upon the container. Things like storing a container in an erasure coded format is much easier once a container is closed. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8741) Proper error msg to be printed when invalid operation type is given to WebHDFS operations.
[
https://issues.apache.org/jira/browse/HDFS-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893757#comment-15893757
]
Surendra Singh Lilhore commented on HDFS-8741:
--
Thanks [~linyiqun] for review..
attached updated patch, please review..
> Proper error msg to be printed when invalid operation type is given to
> WebHDFS operations.
> --
>
> Key: HDFS-8741
> URL: https://issues.apache.org/jira/browse/HDFS-8741
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Attachments: HDFS-8741.001.patch, HDFS-8741.002.patch,
> HDFS-8741.003.patch, HDFS-8741.004.patch
>
>
> When wrong operation type is given to WebHDFS operations, following Error
> message is printed --
> For ex: CREATE is called with GET instead of PUT--
> HTTP/1.1 400 Bad Request
> ..
> {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Invalid
> value for webhdfs parameter \"op\": {color:red}No enum constant
> org.apache.hadoop.hdfs.web.resources.PutOpParam.Op.CREATE"}}{color}
> Expected--
> Valid Error message to be printed
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8741) Proper error msg to be printed when invalid operation type is given to WebHDFS operations.
[
https://issues.apache.org/jira/browse/HDFS-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-8741:
-
Attachment: HDFS-8741.004.patch
> Proper error msg to be printed when invalid operation type is given to
> WebHDFS operations.
> --
>
> Key: HDFS-8741
> URL: https://issues.apache.org/jira/browse/HDFS-8741
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Attachments: HDFS-8741.001.patch, HDFS-8741.002.patch,
> HDFS-8741.003.patch, HDFS-8741.004.patch
>
>
> When wrong operation type is given to WebHDFS operations, following Error
> message is printed --
> For ex: CREATE is called with GET instead of PUT--
> HTTP/1.1 400 Bad Request
> ..
> {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Invalid
> value for webhdfs parameter \"op\": {color:red}No enum constant
> org.apache.hadoop.hdfs.web.resources.PutOpParam.Op.CREATE"}}{color}
> Expected--
> Valid Error message to be printed
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11196) Ozone: Improve logging and error handling in the container layer
[ https://issues.apache.org/jira/browse/HDFS-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11196: Attachment: HDFS-11196-HDFS-7240.002.patch [~xyao] Thanks for the comments. I have addressed all comments in the v2 patch. bq. 1. We seems to have an extra StorageContainerException defined thanks, fixed. bq. 2. NIT: StorageContainerException.java fixed bq. 295: can we remove the TODO i done bq. Line 300: do we need to plumb the traceID removed. bq. Line 167/255: NIT: javadoc throws does fixed one, line 255 not able to find the issue. bq. Line 293: do we need to throw StorageContainerException with CONTAINER_INTERNAL_ERROR not really since we are getting an IO_EXCEPTION from underlying layer. bq. Line 310: NIT: javadoc throws declaration fixed bq. Line 319: can we include the path in the exception message for trouble shooting? fixed. bq. Line 102/107/156/277: throws StorageContainerException fixed in some locations. Line 210: should we rethrow StorageContainerException here? fixed. bq. Maybe adding more specific PUT_SMALLFILE_ERROR/GET_SMALLFILE_ERROR? fixed. > Ozone: Improve logging and error handling in the container layer > > > Key: HDFS-11196 > URL: https://issues.apache.org/jira/browse/HDFS-11196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11196-HDFS-7240.001.patch, > HDFS-11196-HDFS-7240.002.patch > > > Improve logging and error handling in container layer. > * With this change Storage Containers return StorageContainerException. > * Precondition checks fail with a human readable error. > * All failed requests are logged with traceID in the dispatcher. > * Returns proper error codes for corresponding failures. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10506) OIV's ReverseXML processor cannot reconstruct some snapshot details
[ https://issues.apache.org/jira/browse/HDFS-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893753#comment-15893753 ] Akira Ajisaka commented on HDFS-10506: -- Hi [~jojochuang], would you review the patch for branch-2? > OIV's ReverseXML processor cannot reconstruct some snapshot details > --- > > Key: HDFS-10506 > URL: https://issues.apache.org/jira/browse/HDFS-10506 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Colin P. McCabe >Assignee: Akira Ajisaka > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-10506.01.patch, HDFS-10506.02.patch, > HDFS-10506.03.patch, HDFS-10506.04.patch, HDFS-10506-branch-2.01.patch > > > OIV's ReverseXML processor cannot reconstruct some snapshot details. > Specifically, should contain a and field, > but does not. should contain a field. OIV also > needs to be changed to emit these fields into the XML (they are currently > missing). -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11451) Ozone: Add protobuf definitions for container reports
[
https://issues.apache.org/jira/browse/HDFS-11451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893735#comment-15893735
]
Hadoop QA commented on HDFS-11451:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
4s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 39s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}112m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.ozone.web.TestOzoneVolumes |
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
| | hadoop.ozone.web.TestOzoneRestWithMiniCluster |
| | hadoop.ozone.container.common.TestEndPoint |
| | hadoop.ozone.TestContainerOperations |
| | hadoop.ozone.web.client.TestBuckets |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.scm.TestAllocateContainer |
| | hadoop.ozone.web.client.TestVolume |
| | hadoop.ozone.web.TestOzoneWebAccess |
| Timed out junit tests |
org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-11451 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855760/HDFS-11451-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 4b5e956f355e 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / e79f9da |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HDFS-11416) Refactor out system default erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893729#comment-15893729 ] Hudson commented on HDFS-11416: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11336 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11336/]) HDFS-11416. Refactor out system default erasure coding policy. (rakeshr: rev 3749152b661d0359b3b941ab1d17177230f3b8dc) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java > Refactor out system default erasure coding policy > - > > Key: HDFS-11416 > URL: https://issues.apache.org/jira/browse/HDFS-11416 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Labels: hdfs-ec-3.0-nice-to-have > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-11416.001.patch, HDFS-11416.002.patch, > HDFS-11416.003.patch, HDFS-11416.004.patch, HDFS-11416.005.patch, > HDFS-11416.006.patch > > > As discussed on HDFS-7859, the system default EC policy is mostly a relic > from development when the system only supported a single global policy. Now, > we support multiple policies, and the system default policy is mostly used by > tests. > We should refactor to remove this concept. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11416) Refactor out system default erasure coding policy
[
https://issues.apache.org/jira/browse/HDFS-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-11416:
Attachment: HDFS-11416.006.patch
Also, attaching the committed patch, where I've fixed one checkstyle warning -
removed unused {{import org.apache.hadoop.test.GenericTestUtils}}.
> Refactor out system default erasure coding policy
> -
>
> Key: HDFS-11416
> URL: https://issues.apache.org/jira/browse/HDFS-11416
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Labels: hdfs-ec-3.0-nice-to-have
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11416.001.patch, HDFS-11416.002.patch,
> HDFS-11416.003.patch, HDFS-11416.004.patch, HDFS-11416.005.patch,
> HDFS-11416.006.patch
>
>
> As discussed on HDFS-7859, the system default EC policy is mostly a relic
> from development when the system only supported a single global policy. Now,
> we support multiple policies, and the system default policy is mostly used by
> tests.
> We should refactor to remove this concept.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11416) Refactor out system default erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-11416: Resolution: Fixed Fix Version/s: 3.0.0-alpha3 Status: Resolved (was: Patch Available) Thanks [~andrew.wang] for the contribution. Thanks [~jojochuang] for the reviews. Committed to trunk. > Refactor out system default erasure coding policy > - > > Key: HDFS-11416 > URL: https://issues.apache.org/jira/browse/HDFS-11416 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Labels: hdfs-ec-3.0-nice-to-have > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-11416.001.patch, HDFS-11416.002.patch, > HDFS-11416.003.patch, HDFS-11416.004.patch, HDFS-11416.005.patch > > > As discussed on HDFS-7859, the system default EC policy is mostly a relic > from development when the system only supported a single global policy. Now, > we support multiple policies, and the system default policy is mostly used by > tests. > We should refactor to remove this concept. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11336) [SPS]: Remove xAttrs when movements done or SPS disabled
[ https://issues.apache.org/jira/browse/HDFS-11336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanbo Liu updated HDFS-11336: -- Attachment: HDFS-11336-HDFS-10285.004.patch Upload v4 patch to address Uma's comments. > [SPS]: Remove xAttrs when movements done or SPS disabled > > > Key: HDFS-11336 > URL: https://issues.apache.org/jira/browse/HDFS-11336 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > Attachments: HDFS-11336-HDFS-10285.001.patch, > HDFS-11336-HDFS-10285.002.patch, HDFS-11336-HDFS-10285.003.patch, > HDFS-11336-HDFS-10285.004.patch > > > 1. When we finish the movement successfully, we should clean Xattrs. > 2. When we disable SPS dynamically, we should clean Xattrs -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11416) Refactor out system default erasure coding policy
[
https://issues.apache.org/jira/browse/HDFS-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893617#comment-15893617
]
Rakesh R edited comment on HDFS-11416 at 3/3/17 3:19 AM:
-
Thanks [~andrew.wang] for the updates. +1 latest patch looks good to me. I'll
commit this shortly.
+Note:+ Compile errors are due to yarn_ui changes and not related to this patch
{code}
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/src/main/webapp/node_modules/merge-trees/index.js:33
class MergeTrees {
^
Unexpected reserved word
SyntaxError: Unexpected reserved word
{code}
was (Author: rakeshr):
Thanks [~andrew.wang] for the updates. +1 latest patch looks good to me.
+Note:+ Compile errors are due to yarn_ui changes and not related to this patch
{code}
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/src/main/webapp/node_modules/merge-trees/index.js:33
class MergeTrees {
^
Unexpected reserved word
SyntaxError: Unexpected reserved word
{code}
> Refactor out system default erasure coding policy
> -
>
> Key: HDFS-11416
> URL: https://issues.apache.org/jira/browse/HDFS-11416
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-11416.001.patch, HDFS-11416.002.patch,
> HDFS-11416.003.patch, HDFS-11416.004.patch, HDFS-11416.005.patch
>
>
> As discussed on HDFS-7859, the system default EC policy is mostly a relic
> from development when the system only supported a single global policy. Now,
> we support multiple policies, and the system default policy is mostly used by
> tests.
> We should refactor to remove this concept.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-8741) Proper error msg to be printed when invalid operation type is given to WebHDFS operations.
[
https://issues.apache.org/jira/browse/HDFS-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893649#comment-15893649
]
Yiqun Lin edited comment on HDFS-8741 at 3/3/17 3:13 AM:
-
Thanks [~surendrasingh] for the patch. Some comments for your test.
* It would be better to add message into {{fails()}}, like following:
{code}
+ @Test
+ public void testHttpOpParams() {
+try {
+ new PostOpParam("TEST");
+ Assert.fail("Construct the PostOpParam with param vaule 'TEST' should be
failed.");
{code}
* Suggest using {{GenericTestUtils#assertExceptionContains}} verify the message
contained in exception.
was (Author: linyiqun):
Thanks [~surendrasingh] for the patch. Some comments for your test.
* It would be better to add message into {{fails()}}, like following:
{code}
+ @Test
+ public void testHttpOpParams() {
+try {
+ new PostOpParam("TEST");
+ Assert.fail("Construct the PostOpParam with param vaule TEST should be
failed.");
{code}
* Suggest using {{GenericTestUtils#assertExceptionContains}} verify the message
contained in exception.
> Proper error msg to be printed when invalid operation type is given to
> WebHDFS operations.
> --
>
> Key: HDFS-8741
> URL: https://issues.apache.org/jira/browse/HDFS-8741
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Attachments: HDFS-8741.001.patch, HDFS-8741.002.patch,
> HDFS-8741.003.patch
>
>
> When wrong operation type is given to WebHDFS operations, following Error
> message is printed --
> For ex: CREATE is called with GET instead of PUT--
> HTTP/1.1 400 Bad Request
> ..
> {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Invalid
> value for webhdfs parameter \"op\": {color:red}No enum constant
> org.apache.hadoop.hdfs.web.resources.PutOpParam.Op.CREATE"}}{color}
> Expected--
> Valid Error message to be printed
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8741) Proper error msg to be printed when invalid operation type is given to WebHDFS operations.
[
https://issues.apache.org/jira/browse/HDFS-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893649#comment-15893649
]
Yiqun Lin commented on HDFS-8741:
-
Thanks [~surendrasingh] for the patch. Some comments for your test.
* It would be better to add message into {{fails()}}, like following:
{code}
+ @Test
+ public void testHttpOpParams() {
+try {
+ new PostOpParam("TEST");
+ Assert.fail("Construct the PostOpParam with param vaule TEST should be
failed.");
{code}
* Suggest using {{GenericTestUtils#assertExceptionContains}} verify the message
contained in exception.
> Proper error msg to be printed when invalid operation type is given to
> WebHDFS operations.
> --
>
> Key: HDFS-8741
> URL: https://issues.apache.org/jira/browse/HDFS-8741
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Attachments: HDFS-8741.001.patch, HDFS-8741.002.patch,
> HDFS-8741.003.patch
>
>
> When wrong operation type is given to WebHDFS operations, following Error
> message is printed --
> For ex: CREATE is called with GET instead of PUT--
> HTTP/1.1 400 Bad Request
> ..
> {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Invalid
> value for webhdfs parameter \"op\": {color:red}No enum constant
> org.apache.hadoop.hdfs.web.resources.PutOpParam.Op.CREATE"}}{color}
> Expected--
> Valid Error message to be printed
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11476) Fix NPE in FsDatasetImpl#checkAndUpdate
[
https://issues.apache.org/jira/browse/HDFS-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893635#comment-15893635
]
Hadoop QA commented on HDFS-11476:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 81 unchanged - 1 fixed = 82 total (was 82) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 38s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11476 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855747/HDFS-11476.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux fbc0a47cde1a 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11
16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8f4817f |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18529/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18529/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18529/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18529/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically
[jira] [Commented] (HDFS-11476) Fix NPE in FsDatasetImpl#checkAndUpdate
[
https://issues.apache.org/jira/browse/HDFS-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893623#comment-15893623
]
Hadoop QA commented on HDFS-11476:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 81 unchanged - 1 fixed = 82 total (was 82) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 57s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestMaintenanceState |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11476 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855746/HDFS-11476.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 82b43aa981b7 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8f4817f |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18528/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18528/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18528/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18528/console |
|
[jira] [Updated] (HDFS-11451) Ozone: Add protobuf definitions for container reports
[ https://issues.apache.org/jira/browse/HDFS-11451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11451: Attachment: HDFS-11451-HDFS-7240.003.patch Rebase to the top of the tree > Ozone: Add protobuf definitions for container reports > - > > Key: HDFS-11451 > URL: https://issues.apache.org/jira/browse/HDFS-11451 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11451-HDFS-7240.001.patch, > HDFS-11451-HDFS-7240.002.patch, HDFS-11451-HDFS-7240.003.patch > > > Container reports are send from datanodes describing the state of the > container. Since a full container report can be as large as 2 MB, the > container reports are send by datanode only when SCM approves it. > In this change, datanode informs the SCM that it has a container report > ready. SCM based on load will send a command to datanode to send the actual > report. The protobuf classes and plumbing required for that change is part of > this patch. This whole container reports handling will be broken into > multiple JIRAs to make it easy to code review. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11416) Refactor out system default erasure coding policy
[
https://issues.apache.org/jira/browse/HDFS-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893617#comment-15893617
]
Rakesh R commented on HDFS-11416:
-
Thanks [~andrew.wang] for the updates. +1 latest patch looks good to me.
+Note:+ Compile errors are due to yarn_ui changes and not related to this patch
{code}
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/src/main/webapp/node_modules/merge-trees/index.js:33
class MergeTrees {
^
Unexpected reserved word
SyntaxError: Unexpected reserved word
{code}
> Refactor out system default erasure coding policy
> -
>
> Key: HDFS-11416
> URL: https://issues.apache.org/jira/browse/HDFS-11416
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-11416.001.patch, HDFS-11416.002.patch,
> HDFS-11416.003.patch, HDFS-11416.004.patch, HDFS-11416.005.patch
>
>
> As discussed on HDFS-7859, the system default EC policy is mostly a relic
> from development when the system only supported a single global policy. Now,
> we support multiple policies, and the system default policy is mostly used by
> tests.
> We should refactor to remove this concept.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11487) Ozone: Exclude container protobuf files from findbugs check
[ https://issues.apache.org/jira/browse/HDFS-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanbo Liu reassigned HDFS-11487: - Assignee: Yuanbo Liu > Ozone: Exclude container protobuf files from findbugs check > --- > > Key: HDFS-11487 > URL: https://issues.apache.org/jira/browse/HDFS-11487 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Yuanbo Liu > Labels: newbie > Fix For: HDFS-7240 > > > We moved the container protobuf definitions from hdfs project to hdfs-client. > This is to track that we need to add an exclusion to hdfs-client findbugs xml > to ignore those files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8112) Relax permission checking for EC related operations
[
https://issues.apache.org/jira/browse/HDFS-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893590#comment-15893590
]
Kai Zheng commented on HDFS-8112:
-
Thanks [~andrew.wang] for the quick taking on this. The patch looks pretty
good. A little minor towards +1, in addition to the check styles, could you
refine {{testCreationErasureCodingZoneWithInvalidPolicy}} by the way, since we
don't favor {{ErasureCodingZone}} concept any more.
> Relax permission checking for EC related operations
> ---
>
> Key: HDFS-8112
> URL: https://issues.apache.org/jira/browse/HDFS-8112
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Kai Zheng
>Assignee: Andrew Wang
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: hdfs-8112.001.patch, hdfs-8112.002.patch
>
>
> We should allow to enforce authorization policy to protect administration
> operations for EC zone and schemas as such behaviors would impact too much
> for a system.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11451) Ozone: Add protobuf definitions for container reports
[
https://issues.apache.org/jira/browse/HDFS-11451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893582#comment-15893582
]
Hadoop QA commented on HDFS-11451:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} HDFS-11451 does not apply to HDFS-7240. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11451 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855753/HDFS-11451-HDFS-7240.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18530/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Add protobuf definitions for container reports
> -
>
> Key: HDFS-11451
> URL: https://issues.apache.org/jira/browse/HDFS-11451
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-7240
>
> Attachments: HDFS-11451-HDFS-7240.001.patch,
> HDFS-11451-HDFS-7240.002.patch
>
>
> Container reports are send from datanodes describing the state of the
> container. Since a full container report can be as large as 2 MB, the
> container reports are send by datanode only when SCM approves it.
> In this change, datanode informs the SCM that it has a container report
> ready. SCM based on load will send a command to datanode to send the actual
> report. The protobuf classes and plumbing required for that change is part of
> this patch. This whole container reports handling will be broken into
> multiple JIRAs to make it easy to code review.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11451) Ozone: Add protobuf definitions for container reports
[ https://issues.apache.org/jira/browse/HDFS-11451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11451: Attachment: HDFS-11451-HDFS-7240.002.patch > Ozone: Add protobuf definitions for container reports > - > > Key: HDFS-11451 > URL: https://issues.apache.org/jira/browse/HDFS-11451 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11451-HDFS-7240.001.patch, > HDFS-11451-HDFS-7240.002.patch > > > Container reports are send from datanodes describing the state of the > container. Since a full container report can be as large as 2 MB, the > container reports are send by datanode only when SCM approves it. > In this change, datanode informs the SCM that it has a container report > ready. SCM based on load will send a command to datanode to send the actual > report. The protobuf classes and plumbing required for that change is part of > this patch. This whole container reports handling will be broken into > multiple JIRAs to make it easy to code review. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11451) Ozone: Add protobuf definitions for container reports
[ https://issues.apache.org/jira/browse/HDFS-11451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11451: Status: Patch Available (was: Open) > Ozone: Add protobuf definitions for container reports > - > > Key: HDFS-11451 > URL: https://issues.apache.org/jira/browse/HDFS-11451 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11451-HDFS-7240.001.patch, > HDFS-11451-HDFS-7240.002.patch > > > Container reports are send from datanodes describing the state of the > container. Since a full container report can be as large as 2 MB, the > container reports are send by datanode only when SCM approves it. > In this change, datanode informs the SCM that it has a container report > ready. SCM based on load will send a command to datanode to send the actual > report. The protobuf classes and plumbing required for that change is part of > this patch. This whole container reports handling will be broken into > multiple JIRAs to make it easy to code review. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[
https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893555#comment-15893555
]
Arpit Agarwal commented on HDFS-10838:
--
The patch needs a trivial rebase to resolve a conflict in hdfs.proto. Overall
the changes look great. I tried this out in a test cluster and the new column
in the NN web ui is very helpful! Two comments:
# We should use monotonicNow to calculate the time difference for the web UI.
DataNodeInfo has two fields for heartbeat - lastUpdate and lastUpdateMonotonic.
We can use the same approach.
# Also we can add the new field(s) to {{JsonUtil#toJsonMap(DatanodeInfo)}}.
Sorry about the long delay in reviewing this.
> Last full block report received time for each DN should be easily discoverable
> --
>
> Key: HDFS-10838
> URL: https://issues.apache.org/jira/browse/HDFS-10838
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ui
>Reporter: Arpit Agarwal
>Assignee: Surendra Singh Lilhore
> Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch,
> HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch,
> HDFS-10838.005.patch, NN_UI.png, NN_UI_relative_time_minutes.png,
> NN_UI_relative_time.png
>
>
> It should be easy for administrators to discover the time of last full block
> report from each DataNode.
> We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin
> -report}}, or both.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11106) libhdfs++: Some refactoring to better organize files
[
https://issues.apache.org/jira/browse/HDFS-11106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893520#comment-15893520
]
Hadoop QA commented on HDFS-11106:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
4s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
51s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
7s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
47s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
49s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
16s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with
JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:78fc6b6 |
| JIRA Issue | HDFS-11106 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855730/HDFS-11106.HDFS-8707.001.patch
|
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux d477efe3cd39 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / 511d9c5 |
| Default Java | 1.7.0_121 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_121
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 |
| JDK v1.7.0_121 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18525/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18525/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> libhdfs++: Some refactoring to better organize files
>
>
> Key: HDFS-11106
> URL: https://issues.apache.org/jira/browse/HDFS-11106
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments:
[jira] [Comment Edited] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[
https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892877#comment-15892877
]
Arpit Agarwal edited comment on HDFS-10838 at 3/3/17 1:05 AM:
--
-Hi [~surendrasingh], I tried to test this patch and ran into an issue that I
had also hit earlier. The NameNode web UI tabs don't load.-
Never mind, there was something wrong on my end.
was (Author: arpitagarwal):
Hi [~surendrasingh], I tried to test this patch and ran into an issue that I
had also hit earlier. The NameNode web UI tabs don't load.
> Last full block report received time for each DN should be easily discoverable
> --
>
> Key: HDFS-10838
> URL: https://issues.apache.org/jira/browse/HDFS-10838
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ui
>Reporter: Arpit Agarwal
>Assignee: Surendra Singh Lilhore
> Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch,
> HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch,
> HDFS-10838.005.patch, NN_UI.png, NN_UI_relative_time_minutes.png,
> NN_UI_relative_time.png
>
>
> It should be easy for administrators to discover the time of last full block
> report from each DataNode.
> We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin
> -report}}, or both.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11486) Client close() should not fail fast if the last block is being decommissioned
[
https://issues.apache.org/jira/browse/HDFS-11486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893474#comment-15893474
]
Andrew Wang commented on HDFS-11486:
Thanks for investigating this Wei-Chiu,
I think this is an issue with the NN's completeFile handling. The
{{hasMinStorage}} check for LIVE nodes excludes D_I_P nodes. This case doesn't
come up often since min rep is normally set to 1, and operators typically
decommission a rack at a time so there'll always be one replica that's not on a
D_I_P node.
IMO it's still safe to commit/complete a block even when on a D_I_P node. The
DecommissionManager does a final block scan before transitioning a node from
D_I_P to Decommissioned, which will catch any under-replicated blocks like
this. It'd be even better though to add these under-replicated blocks to the
replication queues at close time.
This might also apply to maintenance mode nodes, though I haven't caught up on
that work.
> Client close() should not fail fast if the last block is being decommissioned
> -
>
> Key: HDFS-11486
> URL: https://issues.apache.org/jira/browse/HDFS-11486
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HDF-11486.test.patch
>
>
> If a DFS client closes a file while the last block is being decommissioned,
> the close() may fail if the decommission of the block does not complete in a
> few seconds.
> When a DataNode is being decommissioned, NameNode marks the DN's state as
> DECOMMISSION_INPROGRESS_INPROGRESS, and blocks with replicas on these
> DataNodes become under-replicated immediately. A close() call which attempts
> to complete the last open block will fail if the number of live replicas is
> below minimal replicated factor, due to too many replicas residing on the
> DataNodes.
> The client internally will try to complete the last open block for up to 5
> times by default, which is roughly 12 seconds. After that, close() throws an
> exception like the following, which is typically not handled properly.
> {noformat}
> java.io.IOException: Unable to close file because the last
> blockBP-33575088-10.0.0.200-1488410554081:blk_1073741827_1003 does not have
> enough number of replicas.
> at
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:864)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:827)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:793)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.hadoop.hdfs.TestDecommission.testCloseWhileDecommission(TestDecommission.java:708)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> Once the exception is thrown, the client usually does not attempt to close
> again, so the file remains in open state, and the last block remains in under
> replicated state.
> Subsequently, administrator runs recoverLease tool to salvage the file, but
> the attempt failed because the block remains in under replicated state. It is
> not clear why the block is never replicated though. However, administrators
> think it becomes a corrupt file because the file remains open via fsck
> -openforwrite and the file modification time is hours ago.
> In summary, I do not think close() should fail because the last block is
> being decommissioned. The block has sufficient number replicas, and it's just
> that some replicas are being decommissioned. Decomm should be transparent to
> clients.
> This issue seems to be more prominent on a very large scale cluster, with min
> replication factor set to 2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail:
[jira] [Commented] (HDFS-11477) Simplify file IO profiling configuration
[ https://issues.apache.org/jira/browse/HDFS-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893456#comment-15893456 ] Hudson commented on HDFS-11477: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11335 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11335/]) Revert "HDFS-11477. Simplify file IO profiling configuration. (wang: rev 8f4817f2c5524c1891a3b3b50e396dd4ca294a2d) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestHdfsConfigFields.java * (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ProfilingFileIoEvents.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FileIoProvider.java > Simplify file IO profiling configuration > > > Key: HDFS-11477 > URL: https://issues.apache.org/jira/browse/HDFS-11477 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Attachments: HDFS-11477.000.patch, HDFS-11477.001.patch, > HDFS-11477.002.patch > > > For Profiling FileIO events, there are 2 keys: > - DFS_DATANODE_ENABLE_FILEIO_PROFILING_KEY for enabling the hooks > - DFS_DATANODE_FILEIO_PROFILING_SAMPLING_FRACTION_KEY for setting the > sampling fraction > We can instead have only the sampling fraction key and set it to 0 if we want > to disable profiling. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8112) Relax permission checking for EC related operations
[
https://issues.apache.org/jira/browse/HDFS-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893452#comment-15893452
]
Hadoop QA commented on HDFS-8112:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 5 new + 354 unchanged - 7 fixed = 359 total (was 361) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 58s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}121m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-8112 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855718/hdfs-8112.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f62e9a143d25 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11
16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a97833e |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18523/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18523/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18523/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18523/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Relax permission checking for EC related operations
> ---
>
> Key: HDFS-8112
>
[jira] [Updated] (HDFS-11476) Fix NPE in FsDatasetImpl#checkAndUpdate
[
https://issues.apache.org/jira/browse/HDFS-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HDFS-11476:
-
Attachment: (was: HDFS-11476.003.patch)
> Fix NPE in FsDatasetImpl#checkAndUpdate
> ---
>
> Key: HDFS-11476
> URL: https://issues.apache.org/jira/browse/HDFS-11476
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-11476.000.patch, HDFS-11476.001.patch,
> HDFS-11476.002.patch, HDFS-11476.003.patch
>
>
> diskMetaFile can be null and passed to compareTo which dereferences it,
> causing NPE
> {code}
> // Compare generation stamp
> if (memBlockInfo.getGenerationStamp() != diskGS) {
> File memMetaFile = FsDatasetUtil.getMetaFile(diskFile,
> memBlockInfo.getGenerationStamp());
> if (memMetaFile.exists()) {
> if (memMetaFile.compareTo(diskMetaFile) != 0) {
> LOG.warn("Metadata file in memory "
> + memMetaFile.getAbsolutePath()
> + " does not match file found by scan "
> + (diskMetaFile == null? null:
> diskMetaFile.getAbsolutePath()));
> }
> } else {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11476) Fix NPE in FsDatasetImpl#checkAndUpdate
[
https://issues.apache.org/jira/browse/HDFS-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HDFS-11476:
-
Attachment: HDFS-11476.003.patch
> Fix NPE in FsDatasetImpl#checkAndUpdate
> ---
>
> Key: HDFS-11476
> URL: https://issues.apache.org/jira/browse/HDFS-11476
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-11476.000.patch, HDFS-11476.001.patch,
> HDFS-11476.002.patch, HDFS-11476.003.patch
>
>
> diskMetaFile can be null and passed to compareTo which dereferences it,
> causing NPE
> {code}
> // Compare generation stamp
> if (memBlockInfo.getGenerationStamp() != diskGS) {
> File memMetaFile = FsDatasetUtil.getMetaFile(diskFile,
> memBlockInfo.getGenerationStamp());
> if (memMetaFile.exists()) {
> if (memMetaFile.compareTo(diskMetaFile) != 0) {
> LOG.warn("Metadata file in memory "
> + memMetaFile.getAbsolutePath()
> + " does not match file found by scan "
> + (diskMetaFile == null? null:
> diskMetaFile.getAbsolutePath()));
> }
> } else {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-9388) Refactor decommission related code to support maintenance state for datanodes
[
https://issues.apache.org/jira/browse/HDFS-9388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy reassigned HDFS-9388:
Assignee: Manoj Govindassamy
[~mingma],
How about renaming {{DecommissionManager}} to something like below
* DatanodeMaintenance
* DatanodeService
* DatanodeProvision
Any other better names you can suggest please ?
Also, will take care of comments in the class to be common for DeCom/
MaintState.
> Refactor decommission related code to support maintenance state for datanodes
> -
>
> Key: HDFS-9388
> URL: https://issues.apache.org/jira/browse/HDFS-9388
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Manoj Govindassamy
>
> Lots of code can be shared between the existing decommission functionality
> and to-be-added maintenance state support for datanodes. To make it easier to
> add maintenance state support, let us first modify the existing code to make
> it more general.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10394) move declaration of okhttp version from hdfs-client to hadoop-project POM
[
https://issues.apache.org/jira/browse/HDFS-10394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893408#comment-15893408
]
Hadoop QA commented on HDFS-10394:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 7m
1s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
30s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
18s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 18s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
12s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
38s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10394 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855731/HDFS-10394.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux acd6bb9034a7 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e61491d |
| Default Java | 1.8.0_121 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/18527/artifact/patchprocess/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/18527/artifact/patchprocess/branch-compile-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/18527/artifact/patchprocess/patch-compile-root.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/18527/artifact/patchprocess/patch-compile-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18527/testReport/ |
| modules | C: hadoop-project hadoop-hdfs-project/hadoop-hdfs-client U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18527/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> move declaration of
[jira] [Commented] (HDFS-11477) Simplify file IO profiling configuration
[ https://issues.apache.org/jira/browse/HDFS-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893398#comment-15893398 ] Arpit Agarwal commented on HDFS-11477: -- Jenkins rightly flagged the v2 patch compile error but I looked at the wrong Jenkins output I think. Still doesn't explain why the v1 patch compilation succeeded. > Simplify file IO profiling configuration > > > Key: HDFS-11477 > URL: https://issues.apache.org/jira/browse/HDFS-11477 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Attachments: HDFS-11477.000.patch, HDFS-11477.001.patch, > HDFS-11477.002.patch > > > For Profiling FileIO events, there are 2 keys: > - DFS_DATANODE_ENABLE_FILEIO_PROFILING_KEY for enabling the hooks > - DFS_DATANODE_FILEIO_PROFILING_SAMPLING_FRACTION_KEY for setting the > sampling fraction > We can instead have only the sampling fraction key and set it to 0 if we want > to disable profiling. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11476) Fix NPE in FsDatasetImpl#checkAndUpdate
[
https://issues.apache.org/jira/browse/HDFS-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893396#comment-15893396
]
Xiaobing Zhou commented on HDFS-11476:
--
made some minor changes by v3.
> Fix NPE in FsDatasetImpl#checkAndUpdate
> ---
>
> Key: HDFS-11476
> URL: https://issues.apache.org/jira/browse/HDFS-11476
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-11476.000.patch, HDFS-11476.001.patch,
> HDFS-11476.002.patch, HDFS-11476.003.patch
>
>
> diskMetaFile can be null and passed to compareTo which dereferences it,
> causing NPE
> {code}
> // Compare generation stamp
> if (memBlockInfo.getGenerationStamp() != diskGS) {
> File memMetaFile = FsDatasetUtil.getMetaFile(diskFile,
> memBlockInfo.getGenerationStamp());
> if (memMetaFile.exists()) {
> if (memMetaFile.compareTo(diskMetaFile) != 0) {
> LOG.warn("Metadata file in memory "
> + memMetaFile.getAbsolutePath()
> + " does not match file found by scan "
> + (diskMetaFile == null? null:
> diskMetaFile.getAbsolutePath()));
> }
> } else {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11476) Fix NPE in FsDatasetImpl#checkAndUpdate
[
https://issues.apache.org/jira/browse/HDFS-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HDFS-11476:
-
Attachment: HDFS-11476.003.patch
> Fix NPE in FsDatasetImpl#checkAndUpdate
> ---
>
> Key: HDFS-11476
> URL: https://issues.apache.org/jira/browse/HDFS-11476
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-11476.000.patch, HDFS-11476.001.patch,
> HDFS-11476.002.patch, HDFS-11476.003.patch
>
>
> diskMetaFile can be null and passed to compareTo which dereferences it,
> causing NPE
> {code}
> // Compare generation stamp
> if (memBlockInfo.getGenerationStamp() != diskGS) {
> File memMetaFile = FsDatasetUtil.getMetaFile(diskFile,
> memBlockInfo.getGenerationStamp());
> if (memMetaFile.exists()) {
> if (memMetaFile.compareTo(diskMetaFile) != 0) {
> LOG.warn("Metadata file in memory "
> + memMetaFile.getAbsolutePath()
> + " does not match file found by scan "
> + (diskMetaFile == null? null:
> diskMetaFile.getAbsolutePath()));
> }
> } else {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11477) Simplify file IO profiling configuration
[ https://issues.apache.org/jira/browse/HDFS-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893390#comment-15893390 ] Arpit Agarwal commented on HDFS-11477: -- Yeah thanks for reverting this [~andrew.wang]. > Simplify file IO profiling configuration > > > Key: HDFS-11477 > URL: https://issues.apache.org/jira/browse/HDFS-11477 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Attachments: HDFS-11477.000.patch, HDFS-11477.001.patch, > HDFS-11477.002.patch > > > For Profiling FileIO events, there are 2 keys: > - DFS_DATANODE_ENABLE_FILEIO_PROFILING_KEY for enabling the hooks > - DFS_DATANODE_FILEIO_PROFILING_SAMPLING_FRACTION_KEY for setting the > sampling fraction > We can instead have only the sampling fraction key and set it to 0 if we want > to disable profiling. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[
https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893388#comment-15893388
]
Hadoop QA commented on HDFS-11384:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 449 unchanged - 0 fixed = 451 total (was 449) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
3s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 69m
40s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | org.apache.hadoop.hdfs.server.balancer.Dispatcher$Source.getBlockList()
calls Thread.sleep() with a lock held At Dispatcher.java:lock held At
Dispatcher.java:[line 778] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11384 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855719/HDFS-11384.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux c8f1eb7124c2 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a97833e |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18524/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/18524/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
|
| Test Results |
[jira] [Reopened] (HDFS-11477) Simplify file IO profiling configuration
[ https://issues.apache.org/jira/browse/HDFS-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HDFS-11477: Reopening, I think this broke the build. I saw a DNConf error referring to a FILEIO config key, reverted this and it passed afterwards. Precommit seems pretty messed up right now unfortunately :( > Simplify file IO profiling configuration > > > Key: HDFS-11477 > URL: https://issues.apache.org/jira/browse/HDFS-11477 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Attachments: HDFS-11477.000.patch, HDFS-11477.001.patch, > HDFS-11477.002.patch > > > For Profiling FileIO events, there are 2 keys: > - DFS_DATANODE_ENABLE_FILEIO_PROFILING_KEY for enabling the hooks > - DFS_DATANODE_FILEIO_PROFILING_SAMPLING_FRACTION_KEY for setting the > sampling fraction > We can instead have only the sampling fraction key and set it to 0 if we want > to disable profiling. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11476) Fix NPE in FsDatasetImpl#checkAndUpdate
[
https://issues.apache.org/jira/browse/HDFS-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893360#comment-15893360
]
Hadoop QA commented on HDFS-11476:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 3m
17s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
27s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
29s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
26s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 26s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 81 unchanged - 1 fixed = 82 total (was 82) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
27s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11476 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855738/HDFS-11476.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 376cb2643d52 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e61491d |
| Default Java | 1.8.0_121 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/18526/artifact/patchprocess/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/18526/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/18526/artifact/patchprocess/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/18526/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/18526/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
|
| compile |
[jira] [Commented] (HDFS-11484) Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate
[
https://issues.apache.org/jira/browse/HDFS-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893345#comment-15893345
]
Xiaoyu Yao commented on HDFS-11484:
---
Here is the latest list of ozone test failures.
{code}
Failed junit tests | hadoop.ozone.web.TestOzoneWebAccess
| hadoop.ozone.web.client.TestKeys
| hadoop.ozone.web.TestOzoneRestWithMiniCluster
| hadoop.hdfs.tools.TestDelegationTokenFetcher
| hadoop.ozone.TestContainerOperations
| hadoop.cblock.TestCBlockServer
| hadoop.ozone.web.client.TestVolume
|
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
hadoop.ozone.container.common.impl.TestContainerPersistence
|
hadoop.ozone.container.common.TestDatanodeStateMachine
| hadoop.ozone.web.TestOzoneVolumes
| hadoop.hdfs.tools.TestDFSAdmin
| hadoop.ozone.web.client.TestBuckets
| hadoop.ozone.scm.TestAllocateContainer
Timed out junit tests |
org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainer
{code}
> Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate
>
>
> Key: HDFS-11484
> URL: https://issues.apache.org/jira/browse/HDFS-11484
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Fix For: HDFS-7240
>
> Attachments: HDFS-11484-HDFS-7240.001.patch
>
>
> Need to waitFor the right condition for reliable verification.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11476) Fix NPE in FsDatasetImpl#checkAndUpdate
[
https://issues.apache.org/jira/browse/HDFS-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893335#comment-15893335
]
Xiaobing Zhou commented on HDFS-11476:
--
v2 fixed the check style issues.
> Fix NPE in FsDatasetImpl#checkAndUpdate
> ---
>
> Key: HDFS-11476
> URL: https://issues.apache.org/jira/browse/HDFS-11476
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-11476.000.patch, HDFS-11476.001.patch,
> HDFS-11476.002.patch
>
>
> diskMetaFile can be null and passed to compareTo which dereferences it,
> causing NPE
> {code}
> // Compare generation stamp
> if (memBlockInfo.getGenerationStamp() != diskGS) {
> File memMetaFile = FsDatasetUtil.getMetaFile(diskFile,
> memBlockInfo.getGenerationStamp());
> if (memMetaFile.exists()) {
> if (memMetaFile.compareTo(diskMetaFile) != 0) {
> LOG.warn("Metadata file in memory "
> + memMetaFile.getAbsolutePath()
> + " does not match file found by scan "
> + (diskMetaFile == null? null:
> diskMetaFile.getAbsolutePath()));
> }
> } else {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11476) Fix NPE in FsDatasetImpl#checkAndUpdate
[
https://issues.apache.org/jira/browse/HDFS-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HDFS-11476:
-
Attachment: HDFS-11476.002.patch
> Fix NPE in FsDatasetImpl#checkAndUpdate
> ---
>
> Key: HDFS-11476
> URL: https://issues.apache.org/jira/browse/HDFS-11476
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-11476.000.patch, HDFS-11476.001.patch,
> HDFS-11476.002.patch
>
>
> diskMetaFile can be null and passed to compareTo which dereferences it,
> causing NPE
> {code}
> // Compare generation stamp
> if (memBlockInfo.getGenerationStamp() != diskGS) {
> File memMetaFile = FsDatasetUtil.getMetaFile(diskFile,
> memBlockInfo.getGenerationStamp());
> if (memMetaFile.exists()) {
> if (memMetaFile.compareTo(diskMetaFile) != 0) {
> LOG.warn("Metadata file in memory "
> + memMetaFile.getAbsolutePath()
> + " does not match file found by scan "
> + (diskMetaFile == null? null:
> diskMetaFile.getAbsolutePath()));
> }
> } else {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10394) move declaration of okhttp version from hdfs-client to hadoop-project POM
[ https://issues.apache.org/jira/browse/HDFS-10394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893306#comment-15893306 ] Xiaobing Zhou commented on HDFS-10394: -- I posted the v0 patch to move okhttp version to hadoop-project/pom.xml. > move declaration of okhttp version from hdfs-client to hadoop-project POM > - > > Key: HDFS-10394 > URL: https://issues.apache.org/jira/browse/HDFS-10394 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Xiaobing Zhou >Priority: Minor > Attachments: HDFS-10394.000.patch > > > The POM dependency on okhttp in hadoop-hdfs-client declares its version in > that POM instead. > the root declaration, including version, must go into the > hadoop-project/pom.xml so that its easy to track use and have only one place > if this version were ever to be incremented. As it stands, if any other > module picked up the library, they could adopt a different version. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10394) move declaration of okhttp version from hdfs-client to hadoop-project POM
[ https://issues.apache.org/jira/browse/HDFS-10394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10394: - Status: Patch Available (was: Open) > move declaration of okhttp version from hdfs-client to hadoop-project POM > - > > Key: HDFS-10394 > URL: https://issues.apache.org/jira/browse/HDFS-10394 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Xiaobing Zhou >Priority: Minor > Attachments: HDFS-10394.000.patch > > > The POM dependency on okhttp in hadoop-hdfs-client declares its version in > that POM instead. > the root declaration, including version, must go into the > hadoop-project/pom.xml so that its easy to track use and have only one place > if this version were ever to be incremented. As it stands, if any other > module picked up the library, they could adopt a different version. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10394) move declaration of okhttp version from hdfs-client to hadoop-project POM
[ https://issues.apache.org/jira/browse/HDFS-10394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10394: - Attachment: HDFS-10394.000.patch > move declaration of okhttp version from hdfs-client to hadoop-project POM > - > > Key: HDFS-10394 > URL: https://issues.apache.org/jira/browse/HDFS-10394 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Xiaobing Zhou >Priority: Minor > Attachments: HDFS-10394.000.patch > > > The POM dependency on okhttp in hadoop-hdfs-client declares its version in > that POM instead. > the root declaration, including version, must go into the > hadoop-project/pom.xml so that its easy to track use and have only one place > if this version were ever to be incremented. As it stands, if any other > module picked up the library, they could adopt a different version. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11477) Simplify file IO profiling configuration
[ https://issues.apache.org/jira/browse/HDFS-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893278#comment-15893278 ] Hudson commented on HDFS-11477: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11334 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11334/]) HDFS-11477. Simplify file IO profiling configuration. Contributed by (arp: rev e61491d4769536ce8b77b416005ce531278ce729) * (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestHdfsConfigFields.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ProfilingFileIoEvents.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FileIoProvider.java > Simplify file IO profiling configuration > > > Key: HDFS-11477 > URL: https://issues.apache.org/jira/browse/HDFS-11477 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HDFS-11477.000.patch, HDFS-11477.001.patch, > HDFS-11477.002.patch > > > For Profiling FileIO events, there are 2 keys: > - DFS_DATANODE_ENABLE_FILEIO_PROFILING_KEY for enabling the hooks > - DFS_DATANODE_FILEIO_PROFILING_SAMPLING_FRACTION_KEY for setting the > sampling fraction > We can instead have only the sampling fraction key and set it to 0 if we want > to disable profiling. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11106) libhdfs++: Some refactoring to better organize files
[ https://issues.apache.org/jira/browse/HDFS-11106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11106: --- Attachment: HDFS-11106.HDFS-8707.001.patch New patch (001) ready to review. Just mechanical changes to structure the RPC code. Mostly pulling things out of rpc_engine.h and rpc_connection.cc. > libhdfs++: Some refactoring to better organize files > > > Key: HDFS-11106 > URL: https://issues.apache.org/jira/browse/HDFS-11106 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-11106.HDFS-8707.000.patch, > HDFS-11106.HDFS-8707.001.patch > > > I propose splitting some of the files that have grown wild over time into > files that align with more specific functionality. It's probably best to do > this in a few pieces so it doesn't invalidate anyone's patches in progress. > Here's what I have in mind, looking for feedback if 1) it's not worth doing > for some reason 2) it will break your patch and you'd like this to wait. I'd > also like to consolidate related functions, mostly protobuf helpers, that are > spread around the library into dedicated files. > Targets (can split each into a separate patch): > * (done in patch 000, committed) separate the implementation of operations > from async shim code in files like filesystem.cc (make a > filesystem_shims.cc). The shims are just boilerplate code that only need to > change if the signature of their async counterparts change. > * (done in patch 000, committed) merge base64.cc into util.cc; base64.cc only > contains a single utility function. > * (done in patch 000, committed) rename hdfs_public_api.h/cc to > hdfs_ioservice.h/cc. Originally all of the implementation declarations of > the public API classes like FileSystemImpl were going to live in here. > Currently only the hdfs::IoServiceImpl lives in there and the other Impl > classes have their own dedicated files. > * split hdfs.cc into hdfs.cc and hdfs_ext.cc. Already have a separate > hdfs_ext.h for C bindings for libhdfs++ specific extensions so > implementations of those that live in hdfs.cc would be moved out. Just makes > things a little cleaner. > * split apart various RPC code based on classes. Things like Request and > RpcConnection get defined in rpc_engine.h and then implemented in a handful > of files which get confusing to navigate e.g. why would one expect Request's > implementation to be in rpc_connection.cc. > * Move all of the protobuf<->C++ struct conversion helpers and protobuf wire > serialization/deserialization functions into a single file. Gives us less > protobuf header includes and less accidental duplication of these sorts of > functions. > Like any refactoring some of it comes down to personal preferences. My hope > is that by breaking these into smaller patches/commits relatively fast > forward progress can be made on stuff everyone agrees while things that > people are concerned about can be worked out in a way that satisfies everyone. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11477) Simplify file IO profiling configuration
[ https://issues.apache.org/jira/browse/HDFS-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893248#comment-15893248 ] Arpit Agarwal edited comment on HDFS-11477 at 3/2/17 11:51 PM: --- Thanks [~hanishakoneru]. +1. was (Author: arpitagarwal): Thanks [~hanishakoneru]. +1 pending Jenkins. > Simplify file IO profiling configuration > > > Key: HDFS-11477 > URL: https://issues.apache.org/jira/browse/HDFS-11477 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HDFS-11477.000.patch, HDFS-11477.001.patch, > HDFS-11477.002.patch > > > For Profiling FileIO events, there are 2 keys: > - DFS_DATANODE_ENABLE_FILEIO_PROFILING_KEY for enabling the hooks > - DFS_DATANODE_FILEIO_PROFILING_SAMPLING_FRACTION_KEY for setting the > sampling fraction > We can instead have only the sampling fraction key and set it to 0 if we want > to disable profiling. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11477) Simplify file IO profiling configuration
[ https://issues.apache.org/jira/browse/HDFS-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-11477: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha3 2.9.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-2. Thanks [~hanishakoneru]. > Simplify file IO profiling configuration > > > Key: HDFS-11477 > URL: https://issues.apache.org/jira/browse/HDFS-11477 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HDFS-11477.000.patch, HDFS-11477.001.patch, > HDFS-11477.002.patch > > > For Profiling FileIO events, there are 2 keys: > - DFS_DATANODE_ENABLE_FILEIO_PROFILING_KEY for enabling the hooks > - DFS_DATANODE_FILEIO_PROFILING_SAMPLING_FRACTION_KEY for setting the > sampling fraction > We can instead have only the sampling fraction key and set it to 0 if we want > to disable profiling. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11477) Simplify file IO profiling configuration
[ https://issues.apache.org/jira/browse/HDFS-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-11477: - Summary: Simplify file IO profiling configuration (was: Combine FileIO Profiling Enable and Sampling Fraction Config Key into one) > Simplify file IO profiling configuration > > > Key: HDFS-11477 > URL: https://issues.apache.org/jira/browse/HDFS-11477 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Attachments: HDFS-11477.000.patch, HDFS-11477.001.patch, > HDFS-11477.002.patch > > > For Profiling FileIO events, there are 2 keys: > - DFS_DATANODE_ENABLE_FILEIO_PROFILING_KEY for enabling the hooks > - DFS_DATANODE_FILEIO_PROFILING_SAMPLING_FRACTION_KEY for setting the > sampling fraction > We can instead have only the sampling fraction key and set it to 0 if we want > to disable profiling. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11477) Combine FileIO Profiling Enable and Sampling Fraction Config Key into one
[ https://issues.apache.org/jira/browse/HDFS-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893248#comment-15893248 ] Arpit Agarwal commented on HDFS-11477: -- Thanks [~hanishakoneru]. +1 pending Jenkins. > Combine FileIO Profiling Enable and Sampling Fraction Config Key into one > - > > Key: HDFS-11477 > URL: https://issues.apache.org/jira/browse/HDFS-11477 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Attachments: HDFS-11477.000.patch, HDFS-11477.001.patch, > HDFS-11477.002.patch > > > For Profiling FileIO events, there are 2 keys: > - DFS_DATANODE_ENABLE_FILEIO_PROFILING_KEY for enabling the hooks > - DFS_DATANODE_FILEIO_PROFILING_SAMPLING_FRACTION_KEY for setting the > sampling fraction > We can instead have only the sampling fraction key and set it to 0 if we want > to disable profiling. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11477) Combine FileIO Profiling Enable and Sampling Fraction Config Key into one
[
https://issues.apache.org/jira/browse/HDFS-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893243#comment-15893243
]
Hadoop QA commented on HDFS-11477:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
16s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 12m
3s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 7m
35s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 35s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
0s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 8s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
|
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11477 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855690/HDFS-11477.002.patch |
| Optional Tests | asflicense mvnsite compile javac javadoc mvninstall
unit findbugs checkstyle |
| uname | Linux b04890b55716 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 747bafa |
| Default Java | 1.8.0_121 |
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/18520/artifact/patchprocess/branch-compile-root.txt
|
| findbugs | v3.0.0 |
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/18520/artifact/patchprocess/patch-compile-root.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/18520/artifact/patchprocess/patch-compile-root.txt
|
| unit |
[jira] [Commented] (HDFS-8112) Relax permission checking for EC related operations
[
https://issues.apache.org/jira/browse/HDFS-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893235#comment-15893235
]
Hadoop QA commented on HDFS-8112:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 197 unchanged - 0 fixed = 201 total (was 197) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}100m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-8112 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855700/hdfs-8112.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 23e5f0296da0 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a97833e |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18522/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18522/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18522/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18522/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Relax permission checking for EC related operations
>
[jira] [Commented] (HDFS-11395) RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode
[
https://issues.apache.org/jira/browse/HDFS-11395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893227#comment-15893227
]
Jing Zhao commented on HDFS-11395:
--
bq. in case of non RemoteException from ExecutionException, what should be done
That should be OK. The current retry policies are supposed to handle all
exceptions including non RemoteException. E.g.,
{{FailoverOnNetworkExceptionRetry}} handles {{ConnectException}},
{{EOFException}} etc.
bq. In that case, is it ok to add an additional field to
RetryInvocationHandler#RetryInfo for holding Exception
Yes, sounds good to me.
> RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the
> Exception thrown from NameNode
>
>
> Key: HDFS-11395
> URL: https://issues.apache.org/jira/browse/HDFS-11395
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Reporter: Nandakumar
>Assignee: Nandakumar
> Attachments: HDFS-11395.000.patch, HDFS-11395.001.patch
>
>
> When using RequestHedgingProxyProvider, in case of Exception (like
> FileNotFoundException) from ActiveNameNode,
> {{RequestHedgingProxyProvider#RequestHedgingInvocationHandler.invoke}}
> receives {{ExecutionException}} since we use {{CompletionService}} for the
> call. The ExecutionException is put into a map and wrapped with
> {{MultiException}}.
> So for a FileNotFoundException the client receives
> {{MultiException(Map(ExecutionException(InvocationTargetException(RemoteException(FileNotFoundException)}}
> It will cause problem in clients which are handling RemoteExceptions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11484) Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate
[ https://issues.apache.org/jira/browse/HDFS-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11484: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Target Version/s: HDFS-7240 Status: Resolved (was: Patch Available) [~xyao] Thank you for the contribution. I have committed this to the feature branch. > Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate > > > Key: HDFS-11484 > URL: https://issues.apache.org/jira/browse/HDFS-11484 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Fix For: HDFS-7240 > > Attachments: HDFS-11484-HDFS-7240.001.patch > > > Need to waitFor the right condition for reliable verification. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11441) Add escaping to error messages in web UIs
[
https://issues.apache.org/jira/browse/HDFS-11441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893221#comment-15893221
]
Andrew Wang commented on HDFS-11441:
I checked the failed unit tests, and they're unrelated to this patch. It looks
like branch-2.6 precommit and test suite is pretty broken.
I applied the patch and poked around. Looks like we get double escaping on the
logLevel endpoint:
{noformat}
Submitted Log Name: ;
Log Class: org.apache.commons.logging.impl.Log4JLogger
Submitted Level: ;
Bad Level : ;
Effective level: INFO
{noformat}
I also tried browseDirectory.jsp with a directory named "<>" and "abc" and it
seems to be escaping the entire link:
{noformat}
http://localhost:50075/browseDirectory.jsp?dir=%2F%3C%3E=50070=127.0.0.1:8020;>
http://localhost:50075/browseDirectory.jsp?dir=%2Fabc=50070=127.0.0.1:8020;>abc
{noformat}
Didn't check the others, but I think we need to do manual verification of these
to make sure they're escaping correctly.
> Add escaping to error messages in web UIs
> -
>
> Key: HDFS-11441
> URL: https://issues.apache.org/jira/browse/HDFS-11441
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
>Priority: Minor
> Attachments: HDFS-11441-branch-2.6.patch, HDFS-11441.patch
>
>
> There's a handful of places where web UIs don't escape error messages. We
> should add escaping in these places.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[ https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yunjiong zhao updated HDFS-11384: - Attachment: (was: HDFS-11384.001.patch) > Add option for balancer to disperse getBlocks calls to avoid NameNode's > rpc.CallQueueLength spike > - > > Key: HDFS-11384 > URL: https://issues.apache.org/jira/browse/HDFS-11384 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: balancer.day.png, balancer.week.png, HDFS-11384.001.patch > > > When running balancer on hadoop cluster which have more than 3000 Datanodes > will cause NameNode's rpc.CallQueueLength spike. We observed this situation > could cause Hbase cluster failure due to RegionServer's WAL timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[ https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yunjiong zhao updated HDFS-11384: - Attachment: HDFS-11384.001.patch > Add option for balancer to disperse getBlocks calls to avoid NameNode's > rpc.CallQueueLength spike > - > > Key: HDFS-11384 > URL: https://issues.apache.org/jira/browse/HDFS-11384 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: balancer.day.png, balancer.week.png, > HDFS-11384.001.patch, HDFS-11384.001.patch > > > When running balancer on hadoop cluster which have more than 3000 Datanodes > will cause NameNode's rpc.CallQueueLength spike. We observed this situation > could cause Hbase cluster failure due to RegionServer's WAL timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11416) Refactor out system default erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893190#comment-15893190 ] Andrew Wang commented on HDFS-11416: I have an unused import I think can be removed at commit time, build failures are in javascript? > Refactor out system default erasure coding policy > - > > Key: HDFS-11416 > URL: https://issues.apache.org/jira/browse/HDFS-11416 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11416.001.patch, HDFS-11416.002.patch, > HDFS-11416.003.patch, HDFS-11416.004.patch, HDFS-11416.005.patch > > > As discussed on HDFS-7859, the system default EC policy is mostly a relic > from development when the system only supported a single global policy. Now, > we support multiple policies, and the system default policy is mostly used by > tests. > We should refactor to remove this concept. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11163) Mover should move the file blocks to default storage once policy is unset
[
https://issues.apache.org/jira/browse/HDFS-11163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893184#comment-15893184
]
Hadoop QA commented on HDFS-11163:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
16s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 11m
46s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
17s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 9m
51s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 9m 51s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 51s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 17s{color} | {color:orange} root: The patch generated 7 new + 424 unchanged
- 2 fixed = 431 total (was 426) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 11s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
8s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11163 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855684/HDFS-11163-004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux bdd7a61f1cfc 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 747bafa |
| Default Java | 1.8.0_121 |
| compile |
[jira] [Updated] (HDFS-8112) Relax permission checking for EC related operations
[ https://issues.apache.org/jira/browse/HDFS-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8112: -- Attachment: hdfs-8112.002.patch One more rev attached, changes: * Realized I missed unsetECPolicy, handled now with write perms * Did some refactors/code cleanups in FSDirErasureCodingOp: made some things private, renamed the unprotected getECPolicy method, removed unused hasECPolicy method, inlined getINodesInPath method only used once for permission clarity * Added testing for all EC related methods exposed in HdfsAdmin > Relax permission checking for EC related operations > --- > > Key: HDFS-8112 > URL: https://issues.apache.org/jira/browse/HDFS-8112 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Kai Zheng >Assignee: Andrew Wang >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: hdfs-8112.001.patch, hdfs-8112.002.patch > > > We should allow to enforce authorization policy to protect administration > operations for EC zone and schemas as such behaviors would impact too much > for a system. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11196) Ozone: Improve logging and error handling in the container layer
[ https://issues.apache.org/jira/browse/HDFS-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893179#comment-15893179 ] Xiaoyu Yao commented on HDFS-11196: --- Thanks [~anu] for working on this. The patch looks pretty good to me. It will help us a lot trouble shooting container layer issues in future. I just have a few minor issues below. +1 otherwise. 1. We seems to have an extra StorageContainerException defined in hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/container/common/helpers/StorageContainerException.java. Can we just keep the one in hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/container/common/helpers/StorageContainerException.java only for org.apache.hadoop.scm.container.common.helpers package. 2. NIT: StorageContainerException.java Line 42/63/83: missing javadoc for parameter result 3. ContainerProtocolCall.java Line 295: can we remove the TODO if this patch has taken care of the right exception thrown issue? Ignore this comment if we still have pending work on this. Line 300: do we need to plumb the traceID into the StorageContianerException as described in the ticket? Looks like it is unused here. 4. ChunkUtils.java Line 167/255: NIT: javadoc throws does not match with the code Line 293: do we need to throw StorageContainerException with CONTAINER_INTERNAL_ERROR like Line 214? 6. ContainerUtils.java Line 310: NIT: javadoc throws declaration Line 319: can we include the path in the exception message for trouble shooting? 7. KeyUtils.java Line 67: NIT: @throws StorageContainerException Line 82: can we include the db path and container name in the exception message for trouble shooting? 8. ContainerManagerImpl.java Line 102/107/156/277: throws StorageContainerException Line 210: should we rethrow StorageContainerException here? 9. Dispatcher.java Line 519/561: throws StorageContainerException Line 552: Not sure if PROTOC_DECODING_ERROR is the only IOException in the try block. Can you confirm? Maybe adding more specific PUT_SMALLFILE_ERROR/GET_SMALLFILE_ERROR? 10. KeyManager.java Line 50: throws StorageContainerException > Ozone: Improve logging and error handling in the container layer > > > Key: HDFS-11196 > URL: https://issues.apache.org/jira/browse/HDFS-11196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11196-HDFS-7240.001.patch > > > Improve logging and error handling in container layer. > * With this change Storage Containers return StorageContainerException. > * Precondition checks fail with a human readable error. > * All failed requests are logged with traceID in the dispatcher. > * Returns proper error codes for corresponding failures. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11416) Refactor out system default erasure coding policy
[
https://issues.apache.org/jira/browse/HDFS-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893169#comment-15893169
]
Hadoop QA commented on HDFS-11416:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
45s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 11m
50s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 10m
2s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 2s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 17s{color} | {color:orange} root: The patch generated 1 new + 298 unchanged
- 1 fixed = 299 total (was 299) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
14s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestMaintenanceState |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11416 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855679/HDFS-11416.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux aef40284fdcf 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11
16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 747bafa |
| Default Java | 1.8.0_121 |
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/18517/artifact/patchprocess/branch-compile-root.txt
|
| findbugs | v3.0.0 |
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/18517/artifact/patchprocess/patch-compile-root.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/18517/artifact/patchprocess/patch-compile-root.txt
|
| checkstyle |
[jira] [Commented] (HDFS-11441) Add escaping to error messages in web UIs
[
https://issues.apache.org/jira/browse/HDFS-11441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893166#comment-15893166
]
Hadoop QA commented on HDFS-11441:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s{color} | {color:green} branch-2.6 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
37s{color} | {color:red} root in branch-2.6 failed with JDK v1.8.0_121. {color}
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
46s{color} | {color:red} root in branch-2.6 failed with JDK v1.7.0_121. {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} branch-2.6 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
59s{color} | {color:green} branch-2.6 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s{color} | {color:green} branch-2.6 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
29s{color} | {color:red} hadoop-common-project/hadoop-common in branch-2.6 has
66 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
48s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2.6 has 273
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} branch-2.6 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
3s{color} | {color:green} branch-2.6 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
34s{color} | {color:red} root in the patch failed with JDK v1.8.0_121. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 34s{color}
| {color:red} root in the patch failed with JDK v1.8.0_121. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
45s{color} | {color:red} root in the patch failed with JDK v1.7.0_121. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 45s{color}
| {color:red} root in the patch failed with JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2243 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
58s{color} | {color:red} The patch 178 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
7s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
24s{color} | {color:green} hadoop-common in the patch passed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
20s{color} | {color:green} hadoop-kms in the
[jira] [Commented] (HDFS-11461) DataNode Disk Outlier Detection
[ https://issues.apache.org/jira/browse/HDFS-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893146#comment-15893146 ] Hanisha Koneru commented on HDFS-11461: --- Thank you [~arpitagarwal] for commiting the patch. > DataNode Disk Outlier Detection > --- > > Key: HDFS-11461 > URL: https://issues.apache.org/jira/browse/HDFS-11461 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-11461.000.patch, HDFS-11461.001.patch, > HDFS-11461.002.patch > > > Similar to how DataNodes collect peer performance statistics, we can collect > disk performance statistics per datanode and detect outliers among them, if > any. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11490) Store KMS SSL keystore password in catalina.properties
John Zhuge created HDFS-11490:
-
Summary: Store KMS SSL keystore password in catalina.properties
Key: HDFS-11490
URL: https://issues.apache.org/jira/browse/HDFS-11490
Project: Hadoop HDFS
Issue Type: Improvement
Components: kms
Affects Versions: 2.9.0
Reporter: John Zhuge
Assignee: John Zhuge
Priority: Minor
HADOOP-14083 stores SSL ciphers in catalina.properties. We can do the same for
SSL keystore password, thus no longer need the current {{sed}} method:
{noformat}
# If ssl, the populate the passwords into ssl-server.xml before starting tomcat
if [ ! "${KMS_SSL_KEYSTORE_PASS}" = "" ] || [ ! "${KMS_SSL_TRUSTSTORE_PASS}" =
"" ]; then
# Set a KEYSTORE_PASS if not already set
KMS_SSL_KEYSTORE_PASS=${KMS_SSL_KEYSTORE_PASS:-password}
KMS_SSL_KEYSTORE_PASS_ESCAPED=$(hadoop_escape "$KMS_SSL_KEYSTORE_PASS")
KMS_SSL_TRUSTSTORE_PASS_ESCAPED=$(hadoop_escape "$KMS_SSL_TRUSTSTORE_PASS")
cat ${CATALINA_BASE}/conf/ssl-server.xml.conf \
| sed
's/"_kms_ssl_keystore_pass_"/'"\"${KMS_SSL_KEYSTORE_PASS_ESCAPED}\""'/g' \
| sed
's/"_kms_ssl_truststore_pass_"/'"\"${KMS_SSL_TRUSTSTORE_PASS_ESCAPED}\""'/g' >
${CATALINA_BASE}/conf/ssl-server.xml
fi
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11485) HttpFS should warn about weak SSL ciphers
[
https://issues.apache.org/jira/browse/HDFS-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-11485:
--
Summary: HttpFS should warn about weak SSL ciphers (was: HttpFS should
warn about weak ssl ciphers)
> HttpFS should warn about weak SSL ciphers
> -
>
> Key: HDFS-11485
> URL: https://issues.apache.org/jira/browse/HDFS-11485
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> HDFS-11418 sets a list of default ciphers that contain a few weak ciphers in
> order to maintain backwards compatibility. In addition, users can select weak
> ciphers by env {{HTTPFS_SSL_CIPHERS}}. It'd nice to get warnings about the
> weak ciphers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11489) KMS should warning about weak SSL ciphers
John Zhuge created HDFS-11489:
-
Summary: KMS should warning about weak SSL ciphers
Key: HDFS-11489
URL: https://issues.apache.org/jira/browse/HDFS-11489
Project: Hadoop HDFS
Issue Type: Improvement
Components: kms
Affects Versions: 2.9.0
Reporter: John Zhuge
Assignee: John Zhuge
Priority: Minor
HADOOP-14083 sets a list of default ciphers that contain a few weak ciphers in
order to maintain backwards compatibility. In addition, users can select weak
ciphers by env {{KMS_SSL_CIPHERS}}. It'd nice to get warnings about the weak
ciphers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-8112) Relax permission checking for EC related operations
[ https://issues.apache.org/jira/browse/HDFS-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-8112: - Assignee: Andrew Wang (was: Rakesh R) > Relax permission checking for EC related operations > --- > > Key: HDFS-8112 > URL: https://issues.apache.org/jira/browse/HDFS-8112 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Kai Zheng >Assignee: Andrew Wang >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: hdfs-8112.001.patch > > > We should allow to enforce authorization policy to protect administration > operations for EC zone and schemas as such behaviors would impact too much > for a system. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8112) Relax permission checking for EC related operations
[ https://issues.apache.org/jira/browse/HDFS-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8112: -- Attachment: hdfs-8112.001.patch Hi Rakesh, hope you don't mind but I saw this on the must-do list and thought I'd make a patch, attached. > Relax permission checking for EC related operations > --- > > Key: HDFS-8112 > URL: https://issues.apache.org/jira/browse/HDFS-8112 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Kai Zheng >Assignee: Rakesh R >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: hdfs-8112.001.patch > > > We should allow to enforce authorization policy to protect administration > operations for EC zone and schemas as such behaviors would impact too much > for a system. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8112) Relax permission checking for EC related operations
[ https://issues.apache.org/jira/browse/HDFS-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8112: -- Status: Patch Available (was: Open) > Relax permission checking for EC related operations > --- > > Key: HDFS-8112 > URL: https://issues.apache.org/jira/browse/HDFS-8112 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Kai Zheng >Assignee: Andrew Wang >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: hdfs-8112.001.patch > > > We should allow to enforce authorization policy to protect administration > operations for EC zone and schemas as such behaviors would impact too much > for a system. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11484) Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate
[
https://issues.apache.org/jira/browse/HDFS-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893062#comment-15893062
]
Hadoop QA commented on HDFS-11484:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
58s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
20s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 52s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.TestOzoneWebAccess |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.web.TestOzoneRestWithMiniCluster |
| | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.ozone.TestContainerOperations |
| | hadoop.cblock.TestCBlockServer |
| | hadoop.ozone.web.client.TestVolume |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
| | hadoop.ozone.container.common.TestDatanodeStateMachine |
| | hadoop.ozone.web.TestOzoneVolumes |
| | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.ozone.web.client.TestBuckets |
| | hadoop.ozone.scm.TestAllocateContainer |
| Timed out junit tests |
org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-11484 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855532/HDFS-11484-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 77e019e02ae2 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 02c3506 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18515/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test
[jira] [Commented] (HDFS-11461) DataNode Disk Outlier Detection
[ https://issues.apache.org/jira/browse/HDFS-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893058#comment-15893058 ] Hudson commented on HDFS-11461: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11332 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11332/]) HDFS-11461. DataNode Disk Outlier Detection. Contributed by Hanisha (arp: rev b3ec531f400dd0a6506dc71233d38ae57b764a43) * (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/SlowNodeDetector.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeDiskMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SlowPeerTracker.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodePeerMetrics.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/OutlierDetector.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/metrics/TestDataNodeOutlierDetectionViaMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/metrics/TestSlowNodeDetector.java > DataNode Disk Outlier Detection > --- > > Key: HDFS-11461 > URL: https://issues.apache.org/jira/browse/HDFS-11461 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-11461.000.patch, HDFS-11461.001.patch, > HDFS-11461.002.patch > > > Similar to how DataNodes collect peer performance statistics, we can collect > disk performance statistics per datanode and detect outliers among them, if > any. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11441) Add escaping to error messages in web UIs
[ https://issues.apache.org/jira/browse/HDFS-11441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-11441: -- Attachment: HDFS-11441-branch-2.6.patch Thanks a lot for the review, [~andrew.wang]. Would you mind verifying that the failed tests are unrelated to this patch? I'm having some trouble with my dev environment right now. Also attaching a patch which does the same thing but for branch-2.6 as well, since that branch includes the old JSPs which also need some fixups. No tests are included since this just changes the rendering of some HTML on the Web UIs. > Add escaping to error messages in web UIs > - > > Key: HDFS-11441 > URL: https://issues.apache.org/jira/browse/HDFS-11441 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.8.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Attachments: HDFS-11441-branch-2.6.patch, HDFS-11441.patch > > > There's a handful of places where web UIs don't escape error messages. We > should add escaping in these places. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11476) Fix NPE in FsDatasetImpl#checkAndUpdate
[
https://issues.apache.org/jira/browse/HDFS-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893037#comment-15893037
]
Hadoop QA commented on HDFS-11476:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 81 unchanged - 1 fixed = 84 total (was 82) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 52s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}100m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestSecureNameNode |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11476 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855677/HDFS-11476.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7a8c95072125 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 747bafa |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18516/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18516/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18516/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18516/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was
[jira] [Updated] (HDFS-11488) Fix failure of TestDataNodeVolumeFailureReporting#testVolFailureStatsPreservedOnNNRestart
[
https://issues.apache.org/jira/browse/HDFS-11488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HDFS-11488:
-
Description:
This one fails in a bunch of build.
Error Message
test timed out after 12 milliseconds
Stacktrace
{noformat}
java.lang.Exception: test timed out after 12 milliseconds
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1249)
at java.lang.Thread.join(Thread.java:1323)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.join(BPServiceActor.java:568)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.join(BPOfferService.java:295)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.shutDownAll(BlockPoolManager.java:118)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:2030)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNode(MiniDFSCluster.java:1993)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:1983)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1962)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1936)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1929)
at
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting.tearDown(TestDataNodeVolumeFailureReporting.java:102)
{noformat}
was:
This one fails in a bunch of build.
Error Message
test timed out after 12 milliseconds
Stacktrace
java.lang.Exception: test timed out after 12 milliseconds
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1249)
at java.lang.Thread.join(Thread.java:1323)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.join(BPServiceActor.java:568)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.join(BPOfferService.java:295)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.shutDownAll(BlockPoolManager.java:118)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:2030)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNode(MiniDFSCluster.java:1993)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:1983)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1962)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1936)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1929)
at
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting.tearDown(TestDataNodeVolumeFailureReporting.java:102)
> Fix failure of
> TestDataNodeVolumeFailureReporting#testVolFailureStatsPreservedOnNNRestart
> -
>
> Key: HDFS-11488
> URL: https://issues.apache.org/jira/browse/HDFS-11488
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> This one fails in a bunch of build.
> Error Message
> test timed out after 12 milliseconds
> Stacktrace
> {noformat}
> java.lang.Exception: test timed out after 12 milliseconds
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1249)
> at java.lang.Thread.join(Thread.java:1323)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.join(BPServiceActor.java:568)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.join(BPOfferService.java:295)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.shutDownAll(BlockPoolManager.java:118)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:2030)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNode(MiniDFSCluster.java:1993)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:1983)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1962)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1936)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1929)
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting.tearDown(TestDataNodeVolumeFailureReporting.java:102)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11488) Fix failure of TestDataNodeVolumeFailureReporting#testVolFailureStatsPreservedOnNNRestart
[ https://issues.apache.org/jira/browse/HDFS-11488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-11488: - Description: This one fails in a bunch of build. Error Message test timed out after 12 milliseconds Stacktrace java.lang.Exception: test timed out after 12 milliseconds at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1249) at java.lang.Thread.join(Thread.java:1323) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.join(BPServiceActor.java:568) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.join(BPOfferService.java:295) at org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.shutDownAll(BlockPoolManager.java:118) at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:2030) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNode(MiniDFSCluster.java:1993) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:1983) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1962) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1936) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1929) at org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting.tearDown(TestDataNodeVolumeFailureReporting.java:102) > Fix failure of > TestDataNodeVolumeFailureReporting#testVolFailureStatsPreservedOnNNRestart > - > > Key: HDFS-11488 > URL: https://issues.apache.org/jira/browse/HDFS-11488 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > This one fails in a bunch of build. > Error Message > test timed out after 12 milliseconds > Stacktrace > java.lang.Exception: test timed out after 12 milliseconds > at java.lang.Object.wait(Native Method) > at java.lang.Thread.join(Thread.java:1249) > at java.lang.Thread.join(Thread.java:1323) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.join(BPServiceActor.java:568) > at > org.apache.hadoop.hdfs.server.datanode.BPOfferService.join(BPOfferService.java:295) > at > org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.shutDownAll(BlockPoolManager.java:118) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:2030) > at > org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNode(MiniDFSCluster.java:1993) > at > org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:1983) > at > org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1962) > at > org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1936) > at > org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1929) > at > org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting.tearDown(TestDataNodeVolumeFailureReporting.java:102) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11488) Fix failure of TestDataNodeVolumeFailureReporting#testVolFailureStatsPreservedOnNNRestart
Xiaobing Zhou created HDFS-11488: Summary: Fix failure of TestDataNodeVolumeFailureReporting#testVolFailureStatsPreservedOnNNRestart Key: HDFS-11488 URL: https://issues.apache.org/jira/browse/HDFS-11488 Project: Hadoop HDFS Issue Type: Bug Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8132) Namenode Startup Failing When we add Jcarder.jar in class Path
[
https://issues.apache.org/jira/browse/HDFS-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893017#comment-15893017
]
vijay commented on HDFS-8132:
-
[~rohit12sh], Thank you for helping me get unblocked here. I could get jcarder
to run successfully with -noverify flag.
> Namenode Startup Failing When we add Jcarder.jar in class Path
> --
>
> Key: HDFS-8132
> URL: https://issues.apache.org/jira/browse/HDFS-8132
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>
> *{color:blue}Namenode while Startup Args{color}* ( Just added the jcarder
> args)
> exec /home/hdfs/jdk1.7.0_72/bin/java -Dproc_namenode -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Dhadoop.log.dir=/opt/ClusterSetup/Hadoop2.7/install/hadoop/namenode/logs
> -Dhadoop.log.file=hadoop.log
> -Dhadoop.home.dir=/opt/ClusterSetup/Hadoop2.7/install/hadoop/namenode
> -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,console
> -Djava.library.path=/opt/ClusterSetup/Hadoop2.7/install/hadoop/namenode/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
> {color:red}-javaagent:/opt/Jcarder/jcarder.jar=outputdir=/opt/Jcarder/Output/nn-jcarder{color}
> -Dhadoop.security.logger=INFO,NullAppender
> org.apache.hadoop.hdfs.server.namenode.NameNode
> Setting outputdir to /opt/Jcarder/Output/nn-jcarder
> Starting JCarder (2.0.0/6) agent
> Opening for writing: /opt/Jcarder/Output/nn-jcarder/jcarder_events.db
> Opening for writing: /opt/Jcarder/Output/nn-jcarder/jcarder_contexts.db
> Not instrumenting standard library classes (AWT, Swing, etc.)
> JCarder agent initialized
> *{color:red}ERROR{color}*
> {noformat}
> Exception in thread "main" java.lang.VerifyError: Expecting a stackmap frame
> at branch target 21
> Exception Details:
> Location:
>
> org/apache/hadoop/hdfs/server/namenode/NameNode.createHAState(Lorg/apache/hadoop/hdfs/server/common/HdfsServerConstants$StartupOption;)Lorg/apache/hadoop/hdfs/server/namenode/ha/HAState;
> @4: ifeq
> Reason:
> Expected stackmap frame at this location.
> Bytecode:
> 000: 2ab4 02d2 9900 112b b203 08a5 000a 2bb2
> 010: 030b a600 07b2 030d b0b2 030f b0
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2615)
> at java.lang.Class.getMethod0(Class.java:2856)
> at java.lang.Class.getMethod(Class.java:1668)
> at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11417) Add datanode admin command to get the storage info.
[
https://issues.apache.org/jira/browse/HDFS-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893016#comment-15893016
]
Hadoop QA commented on HDFS-11417:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange} hadoop-hdfs-project: The patch generated 3 new +
357 unchanged - 3 fixed = 360 total (was 360) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
8s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 92m
44s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}137m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11417 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855666/HDFS-11417.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 5aba8c3759ef 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / eeca8b0 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18514/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18514/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output |
[jira] [Commented] (HDFS-11476) Fix NPE in FsDatasetImpl#checkAndUpdate
[
https://issues.apache.org/jira/browse/HDFS-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893004#comment-15893004
]
Arpit Agarwal commented on HDFS-11476:
--
+1 pending Jenkins.
> Fix NPE in FsDatasetImpl#checkAndUpdate
> ---
>
> Key: HDFS-11476
> URL: https://issues.apache.org/jira/browse/HDFS-11476
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-11476.000.patch, HDFS-11476.001.patch
>
>
> diskMetaFile can be null and passed to compareTo which dereferences it,
> causing NPE
> {code}
> // Compare generation stamp
> if (memBlockInfo.getGenerationStamp() != diskGS) {
> File memMetaFile = FsDatasetUtil.getMetaFile(diskFile,
> memBlockInfo.getGenerationStamp());
> if (memMetaFile.exists()) {
> if (memMetaFile.compareTo(diskMetaFile) != 0) {
> LOG.warn("Metadata file in memory "
> + memMetaFile.getAbsolutePath()
> + " does not match file found by scan "
> + (diskMetaFile == null? null:
> diskMetaFile.getAbsolutePath()));
> }
> } else {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11461) DataNode Disk Outlier Detection
[ https://issues.apache.org/jira/browse/HDFS-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-11461: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha3 Status: Resolved (was: Patch Available) I've committed this to trunk. Thanks for the contribution [~hanishakoneru]. > DataNode Disk Outlier Detection > --- > > Key: HDFS-11461 > URL: https://issues.apache.org/jira/browse/HDFS-11461 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru > Fix For: 3.0.0-alpha3 > > Attachments: HDFS-11461.000.patch, HDFS-11461.001.patch, > HDFS-11461.002.patch > > > Similar to how DataNodes collect peer performance statistics, we can collect > disk performance statistics per datanode and detect outliers among them, if > any. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11294) libhdfs++: Segfault in HA failover if DNS lookup for both Namenodes fails
[ https://issues.apache.org/jira/browse/HDFS-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer resolved HDFS-11294. Resolution: Won't Fix Closing, invalid use on my part. > libhdfs++: Segfault in HA failover if DNS lookup for both Namenodes fails > - > > Key: HDFS-11294 > URL: https://issues.apache.org/jira/browse/HDFS-11294 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > > Hit while doing more manual testing on HDFS-11028. > The HANamenodeTracker takes an asio endpoint to figure out what endpoint on > the other node to try next during a failover. This is done by passing the > element at index 0 from endpoints (a std::vector). > When DNS fails the endpoints vector for that node will be empty so the > iterator returned by endpoints\[0\] is just a null pointer that gets > dereferenced and causes a segfault. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11484) Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate
[ https://issues.apache.org/jira/browse/HDFS-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892951#comment-15892951 ] Xiaoyu Yao commented on HDFS-11484: --- Thanks [~anu]! > Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate > > > Key: HDFS-11484 > URL: https://issues.apache.org/jira/browse/HDFS-11484 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11484-HDFS-7240.001.patch > > > Need to waitFor the right condition for reliable verification. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11477) Combine FileIO Profiling Enable and Sampling Fraction Config Key into one
[ https://issues.apache.org/jira/browse/HDFS-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDFS-11477: -- Attachment: HDFS-11477.002.patch Added Log statements in patch v02. > Combine FileIO Profiling Enable and Sampling Fraction Config Key into one > - > > Key: HDFS-11477 > URL: https://issues.apache.org/jira/browse/HDFS-11477 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Attachments: HDFS-11477.000.patch, HDFS-11477.001.patch, > HDFS-11477.002.patch > > > For Profiling FileIO events, there are 2 keys: > - DFS_DATANODE_ENABLE_FILEIO_PROFILING_KEY for enabling the hooks > - DFS_DATANODE_FILEIO_PROFILING_SAMPLING_FRACTION_KEY for setting the > sampling fraction > We can instead have only the sampling fraction key and set it to 0 if we want > to disable profiling. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11163) Mover should move the file blocks to default storage once policy is unset
[
https://issues.apache.org/jira/browse/HDFS-11163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-11163:
--
Attachment: HDFS-11163-004.patch
Thanks [~vinayrpet] for review..
Attached updated patch..
> Mover should move the file blocks to default storage once policy is unset
> -
>
> Key: HDFS-11163
> URL: https://issues.apache.org/jira/browse/HDFS-11163
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.8.0
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-11163-001.patch, HDFS-11163-002.patch,
> HDFS-11163-003.patch, HDFS-11163-004.patch
>
>
> HDFS-9534 added new API in FileSystem to unset the storage policy. Once
> policy is unset blocks should move back to the default storage policy.
> Currently mover is not moving file blocks which have zero storage ID
> {code}
> // currently we ignore files with unspecified storage policy
> if (policyId == HdfsConstants.BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) {
> return;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892889#comment-15892889 ] Andrew Wang commented on HDFS-7285: --- Hi Marcell, I recommend basing the patch off of the "trunk" branch, we've been pretty actively working on EC in preparation for alpha3. I looked at the benchmark results, very interesting! I'm guessing the X axis is the EC cell size? We've chosen the RS parameters for our built-in EC policies somewhat arbitrarily, but maybe we should tune them for better performance. Are there any patterns you see wrt the RS parameters? It looks like the #1 factor is how many parity blocks there are, though there's some weirdness like 10+4 being faster than 9+4. There seem to be differences based on the cell size, but no pattern I can see. Another question, is the Chocolate Cloud RS implementation byte-for-byte compatible with the Hadoop ISA-L and Java RS coder implementations? If not, it'll be harder for users to experiment with CC RS since it'll require rewriting data. We'll also need to add a new set of EC policies, which increases the configuration and maintenance overhead. In this case, I'd like to see more clear wins and a stable native code release before we put this in a release. In the meanwhile, we can also set up a development branch so you can work on the Hadoop integration. We've also made some progress toward pluggable EC policies and coder framework, though it's not done yet. If this gets finished, it'd be a nice way of making your RS implementation accessible to users without requiring it to be shipped with Hadoop itself. > Erasure Coding Support inside HDFS > -- > > Key: HDFS-7285 > URL: https://issues.apache.org/jira/browse/HDFS-7285 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Weihua Jiang >Assignee: Zhe Zhang > Fix For: 3.0.0-alpha1 > > Attachments: Compare-consolidated-20150824.diff, > Consolidated-20150707.patch, Consolidated-20150806.patch, > Consolidated-20150810.patch, ECAnalyzer.py, ECParser.py, > fsimage-analysis-20150105.pdf, HDFS-7285-Consolidated-20150911.patch, > HDFS-7285-initial-PoC.patch, HDFS-7285-merge-consolidated-01.patch, > HDFS-7285-merge-consolidated-trunk-01.patch, > HDFS-7285-merge-consolidated.trunk.03.patch, > HDFS-7285-merge-consolidated.trunk.04.patch, HDFS-bistriped.patch, > HDFS-EC-merge-consolidated-01.patch, HDFS-EC-Merge-PoC-20150624.patch, > HDFSErasureCodingDesign-20141028.pdf, HDFSErasureCodingDesign-20141217.pdf, > HDFSErasureCodingDesign-20150204.pdf, HDFSErasureCodingDesign-20150206.pdf, > HDFSErasureCodingPhaseITestPlan.pdf, > HDFSErasureCodingSystemTestPlan-20150824.pdf, > HDFSErasureCodingSystemTestReport-20150826.pdf > > > Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice > of data reliability, comparing to the existing HDFS 3-replica approach. For > example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, > with storage overhead only being 40%. This makes EC a quite attractive > alternative for big data storage, particularly for cold data. > Facebook had a related open source project called HDFS-RAID. It used to be > one of the contribute packages in HDFS but had been removed since Hadoop 2.0 > for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends > on MapReduce to do encoding and decoding tasks; 2) it can only be used for > cold files that are intended not to be appended anymore; 3) the pure Java EC > coding implementation is extremely slow in practical use. Due to these, it > might not be a good idea to just bring HDFS-RAID back. > We (Intel and Cloudera) are working on a design to build EC into HDFS that > gets rid of any external dependencies, makes it self-contained and > independently maintained. This design lays the EC feature on the storage type > support and considers compatible with existing HDFS features like caching, > snapshot, encryption, high availability and etc. This design will also > support different EC coding schemes, implementations and policies for > different deployment scenarios. By utilizing advanced libraries (e.g. Intel > ISA-L library), an implementation can greatly improve the performance of EC > encoding/decoding and makes the EC solution even more attractive. We will > post the design document soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8498) Blocks can be committed with wrong size
[
https://issues.apache.org/jira/browse/HDFS-8498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892887#comment-15892887
]
Hadoop QA commented on HDFS-8498:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
35s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 113 unchanged - 6 fixed = 117 total (was 119) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3239 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 1m
21s{color} | {color:red} The patch 77 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 43m 43s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_121. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
19s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_121 Failed junit tests | hadoop.hdfs.TestHFlush |
| | hadoop.hdfs.server.namenode.TestAddBlock |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots |
| | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.web.TestHttpsFileSystem |
| |
[jira] [Commented] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[
https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892884#comment-15892884
]
Hadoop QA commented on HDFS-10838:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-10838 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-10838 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840563/HDFS-10838.005.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18518/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Last full block report received time for each DN should be easily discoverable
> --
>
> Key: HDFS-10838
> URL: https://issues.apache.org/jira/browse/HDFS-10838
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ui
>Reporter: Arpit Agarwal
>Assignee: Surendra Singh Lilhore
> Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch,
> HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch,
> HDFS-10838.005.patch, NN_UI.png, NN_UI_relative_time_minutes.png,
> NN_UI_relative_time.png
>
>
> It should be easy for administrators to discover the time of last full block
> report from each DataNode.
> We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin
> -report}}, or both.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[
https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892877#comment-15892877
]
Arpit Agarwal commented on HDFS-10838:
--
Hi [~surendrasingh], I tried to test this patch and ran into an issue that I
had also hit earlier. The NameNode web UI tabs don't load.
> Last full block report received time for each DN should be easily discoverable
> --
>
> Key: HDFS-10838
> URL: https://issues.apache.org/jira/browse/HDFS-10838
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ui
>Reporter: Arpit Agarwal
>Assignee: Surendra Singh Lilhore
> Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch,
> HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch,
> HDFS-10838.005.patch, NN_UI.png, NN_UI_relative_time_minutes.png,
> NN_UI_relative_time.png
>
>
> It should be easy for administrators to discover the time of last full block
> report from each DataNode.
> We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin
> -report}}, or both.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11416) Refactor out system default erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11416: --- Attachment: HDFS-11416.005.patch Thanks for reviewing Rakesh, new patch attached to address your review comments. > Refactor out system default erasure coding policy > - > > Key: HDFS-11416 > URL: https://issues.apache.org/jira/browse/HDFS-11416 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11416.001.patch, HDFS-11416.002.patch, > HDFS-11416.003.patch, HDFS-11416.004.patch, HDFS-11416.005.patch > > > As discussed on HDFS-7859, the system default EC policy is mostly a relic > from development when the system only supported a single global policy. Now, > we support multiple policies, and the system default policy is mostly used by > tests. > We should refactor to remove this concept. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11487) Ozone: Exclude container protobuf files from findbugs check
[ https://issues.apache.org/jira/browse/HDFS-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11487: Labels: newbie (was: ) > Ozone: Exclude container protobuf files from findbugs check > --- > > Key: HDFS-11487 > URL: https://issues.apache.org/jira/browse/HDFS-11487 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer > Labels: newbie > Fix For: HDFS-7240 > > > We moved the container protobuf definitions from hdfs project to hdfs-client. > This is to track that we need to add an exclusion to hdfs-client findbugs xml > to ignore those files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11487) Ozone: Exclude container protobuf files from findbugs check
[ https://issues.apache.org/jira/browse/HDFS-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892858#comment-15892858 ] Anu Engineer commented on HDFS-11487: - Very similar to the fix in https://issues.apache.org/jira/browse/HDFS-11473 > Ozone: Exclude container protobuf files from findbugs check > --- > > Key: HDFS-11487 > URL: https://issues.apache.org/jira/browse/HDFS-11487 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer > Labels: newbie > Fix For: HDFS-7240 > > > We moved the container protobuf definitions from hdfs project to hdfs-client. > This is to track that we need to add an exclusion to hdfs-client findbugs xml > to ignore those files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11476) Fix NPE in FsDatasetImpl#checkAndUpdate
[
https://issues.apache.org/jira/browse/HDFS-11476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892853#comment-15892853
]
Xiaobing Zhou commented on HDFS-11476:
--
Posted v1 to fix it. Thanks for reviews. [~arpitagarwal]
> Fix NPE in FsDatasetImpl#checkAndUpdate
> ---
>
> Key: HDFS-11476
> URL: https://issues.apache.org/jira/browse/HDFS-11476
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-11476.000.patch, HDFS-11476.001.patch
>
>
> diskMetaFile can be null and passed to compareTo which dereferences it,
> causing NPE
> {code}
> // Compare generation stamp
> if (memBlockInfo.getGenerationStamp() != diskGS) {
> File memMetaFile = FsDatasetUtil.getMetaFile(diskFile,
> memBlockInfo.getGenerationStamp());
> if (memMetaFile.exists()) {
> if (memMetaFile.compareTo(diskMetaFile) != 0) {
> LOG.warn("Metadata file in memory "
> + memMetaFile.getAbsolutePath()
> + " does not match file found by scan "
> + (diskMetaFile == null? null:
> diskMetaFile.getAbsolutePath()));
> }
> } else {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
