[jira] [Updated] (HDFS-13484) RBF: Disable Nameservices from the federation
[ https://issues.apache.org/jira/browse/HDFS-13484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13484: --- Attachment: HDFS-13484.001.patch > RBF: Disable Nameservices from the federation > - > > Key: HDFS-13484 > URL: https://issues.apache.org/jira/browse/HDFS-13484 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13484.000.patch, HDFS-13484.001.patch > > > HDFS-13478 introduced the Decommission store. We should disable the access to > decommissioned subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13478) RBF: Disabled Nameservice store API
[ https://issues.apache.org/jira/browse/HDFS-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446644#comment-16446644 ] Hudson commented on HDFS-13478: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14042 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14042/]) HDFS-13478. RBF: Disabled Nameservice store API. Contributed by Inigo (yqlin: rev b6dae26f4490bbd03360b8df19ef00cf843430f3) * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/impl/DisabledNameserviceStoreImpl.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/GetDisabledNameservicesRequest.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/NameserviceManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/proto/FederationProtocol.proto * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdmin.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/DisabledNameserviceStore.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClient.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/DisableNameserviceResponsePBImpl.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/EnableNameserviceResponsePBImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/DisableNameserviceResponse.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/store/driver/TestStateStoreDriverBase.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/GetDisabledNameservicesResponse.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/GetDisabledNameservicesRequestPBImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/proto/RouterProtocol.proto * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/EnableNameserviceRequest.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/GetDisabledNameservicesResponsePBImpl.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/EnableNameserviceResponse.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/EnableNameserviceRequestPBImpl.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/impl/pb/DisabledNameservicePBImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/protocolPB/RouterAdminProtocolServerSideTranslatorPB.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/BaseRecord.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/DisableNameserviceRequestPBImpl.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/DisableNameserviceRequest.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/StateStoreService.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/protocolPB/RouterAdminProtocolTranslatorPB.java * (add) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/DisabledNameservice.java > RBF: Disabled Nameservice store API > --- > > Key: HDFS-13478 > URL: https://issues.apache.org/jira/browse/HDFS-13478 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13478.000.patch, HDFS-13478.001.patch, > HDFS-13478.002.patch, HDFS-13478.003.patc
[jira] [Commented] (HDFS-13478) RBF: Disabled Nameservice store API
[ https://issues.apache.org/jira/browse/HDFS-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446636#comment-16446636 ] Yiqun Lin commented on HDFS-13478: -- LGTM, +1. Committing this. > RBF: Disabled Nameservice store API > --- > > Key: HDFS-13478 > URL: https://issues.apache.org/jira/browse/HDFS-13478 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13478.000.patch, HDFS-13478.001.patch, > HDFS-13478.002.patch, HDFS-13478.003.patch, HDFS-13478.004.patch, > HDFS-13478.005.patch > > > We have a subcluster in our federation that is for testing and is > missbehaving. This has a negative impact on the performance with operations > that go to every subcluster (e.g., renewLease() or setSafeMode()). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10183) Prevent race condition during class initialization
[
https://issues.apache.org/jira/browse/HDFS-10183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated HDFS-10183:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.9.2
3.1.1
3.2.0
2.8.4
Status: Resolved (was: Patch Available)
Committed. Thanks [~pavgustinov] for your contribution!
> Prevent race condition during class initialization
> --
>
> Key: HDFS-10183
> URL: https://issues.apache.org/jira/browse/HDFS-10183
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.9.0
>Reporter: Pavel Avgustinov
>Assignee: Pavel Avgustinov
>Priority: Minor
> Fix For: 2.8.4, 3.2.0, 3.1.1, 2.9.2
>
> Attachments: HADOOP-12944.1.patch, HDFS-10183.2.patch
>
>
> In HADOOP-11969, [~busbey] tracked down a non-deterministic
> {{NullPointerException}} to an oddity in the Java memory model: When multiple
> threads trigger the loading of a class at the same time, one of them wins and
> creates the {{java.lang.Class}} instance; the others block during this
> initialization, but once it is complete they may obtain a reference to the
> {{Class}} which has non-{{final}} fields still containing their default (i.e.
> {{null}}) values. This leads to runtime failures that are hard to debug or
> diagnose.
> HADOOP-11969 observed that {{ThreadLocal}} fields, by their very nature, are
> very likely to be accessed from multiple threads, and thus the problem is
> particularly severe there. Consequently, the patch removed all occurrences of
> the issue in the code base.
> Unfortunately, since then HDFS-7964 has [reverted one of the fixes during a
> refactoring|https://github.com/apache/hadoop/commit/2151716832ad14932dd65b1a4e47e64d8d6cd767#diff-0c2e9f7f9e685f38d1a11373b627cfa6R151],
> and introduced a [new instance of the
> problem|https://github.com/apache/hadoop/commit/2151716832ad14932dd65b1a4e47e64d8d6cd767#diff-6334d0df7d9aefbccd12b21bb7603169R43].
> The attached patch addresses the issue by adding the missing {{final}}
> modifier in these two cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10183) Prevent race condition during class initialization
[
https://issues.apache.org/jira/browse/HDFS-10183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446618#comment-16446618
]
Hudson commented on HDFS-10183:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14041 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14041/])
HDFS-10183. Prevent race condition during class initialization. (sjlee: rev
f40969a141ec6aff254c41e4185cc61ea9e4e554)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogAsync.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
> Prevent race condition during class initialization
> --
>
> Key: HDFS-10183
> URL: https://issues.apache.org/jira/browse/HDFS-10183
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.9.0
>Reporter: Pavel Avgustinov
>Assignee: Pavel Avgustinov
>Priority: Minor
> Attachments: HADOOP-12944.1.patch, HDFS-10183.2.patch
>
>
> In HADOOP-11969, [~busbey] tracked down a non-deterministic
> {{NullPointerException}} to an oddity in the Java memory model: When multiple
> threads trigger the loading of a class at the same time, one of them wins and
> creates the {{java.lang.Class}} instance; the others block during this
> initialization, but once it is complete they may obtain a reference to the
> {{Class}} which has non-{{final}} fields still containing their default (i.e.
> {{null}}) values. This leads to runtime failures that are hard to debug or
> diagnose.
> HADOOP-11969 observed that {{ThreadLocal}} fields, by their very nature, are
> very likely to be accessed from multiple threads, and thus the problem is
> particularly severe there. Consequently, the patch removed all occurrences of
> the issue in the code base.
> Unfortunately, since then HDFS-7964 has [reverted one of the fixes during a
> refactoring|https://github.com/apache/hadoop/commit/2151716832ad14932dd65b1a4e47e64d8d6cd767#diff-0c2e9f7f9e685f38d1a11373b627cfa6R151],
> and introduced a [new instance of the
> problem|https://github.com/apache/hadoop/commit/2151716832ad14932dd65b1a4e47e64d8d6cd767#diff-6334d0df7d9aefbccd12b21bb7603169R43].
> The attached patch addresses the issue by adding the missing {{final}}
> modifier in these two cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13217) Log audit event only used last EC policy name when add multiple policies from file
[
https://issues.apache.org/jira/browse/HDFS-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446615#comment-16446615
]
Xiao Chen commented on HDFS-13217:
--
Thanks for the work here [~liaoyuxiangqin].
Minor but the {{responses}} and {{addECPolicyNames}} arraylists could have been
preallocated with {{policies.length}}, like {{new
ArrayList<>(policies.length);}}
> Log audit event only used last EC policy name when add multiple policies from
> file
> ---
>
> Key: HDFS-13217
> URL: https://issues.apache.org/jira/browse/HDFS-13217
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Major
> Attachments: HDFS-13217.001.patch, HDFS-13217.002.patch,
> HDFS-13217.003.patch
>
>
> When i read the addErasureCodingPolicies() of FSNamesystem class in namenode,
> i found the following code only used last ec policy name for logAuditEvent,
> i think this audit log can't track whole policies for the add multiple
> erasure coding policies to the ErasureCodingPolicyManager. Thanks.
> {code:java|title=FSNamesystem.java|borderStyle=solid}
> try {
> checkOperation(OperationCategory.WRITE);
> checkNameNodeSafeMode("Cannot add erasure coding policy");
> for (ErasureCodingPolicy policy : policies) {
> try {
> ErasureCodingPolicy newPolicy =
> FSDirErasureCodingOp.addErasureCodingPolicy(this, policy,
> logRetryCache);
> addECPolicyName = newPolicy.getName();
> responses.add(new AddErasureCodingPolicyResponse(newPolicy));
> } catch (HadoopIllegalArgumentException e) {
> responses.add(new AddErasureCodingPolicyResponse(policy, e));
> }
> }
> success = true;
> return responses.toArray(new AddErasureCodingPolicyResponse[0]);
> } finally {
> writeUnlock(operationName);
> if (success) {
> getEditLog().logSync();
> }
> logAuditEvent(success, operationName,addECPolicyName, null, null);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13488) RBF: Reject requests when a Router is overloaded
[
https://issues.apache.org/jira/browse/HDFS-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446605#comment-16446605
]
genericqa commented on HDFS-13488:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 46s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m
1s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13488 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920127/HDFS-13488.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux a5cfe11b294d 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 28e2244 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24026/testReport/ |
| Max. process+thread count | 1335 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24026/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
>
[jira] [Commented] (HDFS-13490) RBF: Fix setSafeMode in the Router
[
https://issues.apache.org/jira/browse/HDFS-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446601#comment-16446601
]
genericqa commented on HDFS-13490:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 27s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
10s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13490 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920124/HDFS-13490.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 20ed6af1ca0e 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 28e2244 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24025/testReport/ |
| Max. process+thread count | 1351 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24025/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> RBF: Fix setSafeMode in the Router
> --
>
> Key: HDFS-13490
> URL: https://issues.apache.
[jira] [Commented] (HDFS-10183) Prevent race condition during class initialization
[
https://issues.apache.org/jira/browse/HDFS-10183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446595#comment-16446595
]
Sangjin Lee commented on HDFS-10183:
Sounds good. I'll commit it to various live branches tonight.
> Prevent race condition during class initialization
> --
>
> Key: HDFS-10183
> URL: https://issues.apache.org/jira/browse/HDFS-10183
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.9.0
>Reporter: Pavel Avgustinov
>Assignee: Pavel Avgustinov
>Priority: Minor
> Attachments: HADOOP-12944.1.patch, HDFS-10183.2.patch
>
>
> In HADOOP-11969, [~busbey] tracked down a non-deterministic
> {{NullPointerException}} to an oddity in the Java memory model: When multiple
> threads trigger the loading of a class at the same time, one of them wins and
> creates the {{java.lang.Class}} instance; the others block during this
> initialization, but once it is complete they may obtain a reference to the
> {{Class}} which has non-{{final}} fields still containing their default (i.e.
> {{null}}) values. This leads to runtime failures that are hard to debug or
> diagnose.
> HADOOP-11969 observed that {{ThreadLocal}} fields, by their very nature, are
> very likely to be accessed from multiple threads, and thus the problem is
> particularly severe there. Consequently, the patch removed all occurrences of
> the issue in the code base.
> Unfortunately, since then HDFS-7964 has [reverted one of the fixes during a
> refactoring|https://github.com/apache/hadoop/commit/2151716832ad14932dd65b1a4e47e64d8d6cd767#diff-0c2e9f7f9e685f38d1a11373b627cfa6R151],
> and introduced a [new instance of the
> problem|https://github.com/apache/hadoop/commit/2151716832ad14932dd65b1a4e47e64d8d6cd767#diff-6334d0df7d9aefbccd12b21bb7603169R43].
> The attached patch addresses the issue by adding the missing {{final}}
> modifier in these two cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10183) Prevent race condition during class initialization
[
https://issues.apache.org/jira/browse/HDFS-10183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446591#comment-16446591
]
Sean Busbey commented on HDFS-10183:
I'm still +1. I'd prefer it go in any branch that's still having releases and
it applies to. I'm willing to help with backports if it's a limiting factor.
> Prevent race condition during class initialization
> --
>
> Key: HDFS-10183
> URL: https://issues.apache.org/jira/browse/HDFS-10183
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.9.0
>Reporter: Pavel Avgustinov
>Assignee: Pavel Avgustinov
>Priority: Minor
> Attachments: HADOOP-12944.1.patch, HDFS-10183.2.patch
>
>
> In HADOOP-11969, [~busbey] tracked down a non-deterministic
> {{NullPointerException}} to an oddity in the Java memory model: When multiple
> threads trigger the loading of a class at the same time, one of them wins and
> creates the {{java.lang.Class}} instance; the others block during this
> initialization, but once it is complete they may obtain a reference to the
> {{Class}} which has non-{{final}} fields still containing their default (i.e.
> {{null}}) values. This leads to runtime failures that are hard to debug or
> diagnose.
> HADOOP-11969 observed that {{ThreadLocal}} fields, by their very nature, are
> very likely to be accessed from multiple threads, and thus the problem is
> particularly severe there. Consequently, the patch removed all occurrences of
> the issue in the code base.
> Unfortunately, since then HDFS-7964 has [reverted one of the fixes during a
> refactoring|https://github.com/apache/hadoop/commit/2151716832ad14932dd65b1a4e47e64d8d6cd767#diff-0c2e9f7f9e685f38d1a11373b627cfa6R151],
> and introduced a [new instance of the
> problem|https://github.com/apache/hadoop/commit/2151716832ad14932dd65b1a4e47e64d8d6cd767#diff-6334d0df7d9aefbccd12b21bb7603169R43].
> The attached patch addresses the issue by adding the missing {{final}}
> modifier in these two cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13468) Add erasure coding metrics into ReadStatistics
[
https://issues.apache.org/jira/browse/HDFS-13468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446589#comment-16446589
]
genericqa commented on HDFS-13468:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
52s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
37s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}219m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
|
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.client.impl.TestBlockReaderLocal |
| | hadoop.hdfs.TestDFSClientRetries |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13468 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920107/HDFS-13468.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux fb1de8baef56 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 28e2244 |
| maven | vers
[jira] [Commented] (HDFS-13408) MiniDFSCluster to support being built on randomized base directory
[ https://issues.apache.org/jira/browse/HDFS-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446582#comment-16446582 ] Íñigo Goiri commented on HDFS-13408: The failed unit tests are unrelated. +1 on [^HDFS-13408.004.patch]. Can we commit this all the way to 2.9? > MiniDFSCluster to support being built on randomized base directory > -- > > Key: HDFS-13408 > URL: https://issues.apache.org/jira/browse/HDFS-13408 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Xiao Liang >Assignee: Xiao Liang >Priority: Major > Labels: windows > Attachments: HDFS-13408.000.patch, HDFS-13408.001.patch, > HDFS-13408.002.patch, HDFS-13408.003.patch, HDFS-13408.004.patch > > > Generated files of MiniDFSCluster during test are not properly cleaned in > Windows, which fails all subsequent test cases using the same default > directory (Windows does not allow other processes to delete them). By > migrating to randomized base directories, the conflict of test path of test > cases will be avoided, even if they are running at the same time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13488) RBF: Reject requests when a Router is overloaded
[ https://issues.apache.org/jira/browse/HDFS-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13488: --- Attachment: HDFS-13488.001.patch > RBF: Reject requests when a Router is overloaded > > > Key: HDFS-13488 > URL: https://issues.apache.org/jira/browse/HDFS-13488 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13488.000.patch, HDFS-13488.001.patch > > > A Router might be overloaded when handling special cases (e.g. a slow > subcluster). The Router could reject the requests and the client could try > with another Router. We should leverage the Standby mechanism for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13490) RBF: Fix setSafeMode in the Router
[ https://issues.apache.org/jira/browse/HDFS-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13490: --- Assignee: Íñigo Goiri Status: Patch Available (was: Open) > RBF: Fix setSafeMode in the Router > -- > > Key: HDFS-13490 > URL: https://issues.apache.org/jira/browse/HDFS-13490 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13490.000.patch > > > RouterRpcServer doesn't handle the isChecked parameter correctly when > forwarding setSafeMode to the namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13490) RBF: Fix setSafeMode in the Router
[ https://issues.apache.org/jira/browse/HDFS-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446577#comment-16446577 ] Íñigo Goiri commented on HDFS-13490: We might be able to put the unit test in some other test that already uses HA for the subclusters. > RBF: Fix setSafeMode in the Router > -- > > Key: HDFS-13490 > URL: https://issues.apache.org/jira/browse/HDFS-13490 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Priority: Major > Attachments: HDFS-13490.000.patch > > > RouterRpcServer doesn't handle the isChecked parameter correctly when > forwarding setSafeMode to the namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13490) RBF: Fix setSafeMode in the Router
[ https://issues.apache.org/jira/browse/HDFS-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13490: --- Attachment: HDFS-13490.000.patch > RBF: Fix setSafeMode in the Router > -- > > Key: HDFS-13490 > URL: https://issues.apache.org/jira/browse/HDFS-13490 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Priority: Major > Attachments: HDFS-13490.000.patch > > > RouterRpcServer doesn't handle the isChecked parameter correctly when > forwarding setSafeMode to the namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13490) RBF: Fix setSafeMode in the Router
Íñigo Goiri created HDFS-13490: -- Summary: RBF: Fix setSafeMode in the Router Key: HDFS-13490 URL: https://issues.apache.org/jira/browse/HDFS-13490 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Íñigo Goiri RouterRpcServer doesn't handle the isChecked parameter correctly when forwarding setSafeMode to the namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13408) MiniDFSCluster to support being built on randomized base directory
[
https://issues.apache.org/jira/browse/HDFS-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446574#comment-16446574
]
genericqa commented on HDFS-13408:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}106m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13408 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920110/HDFS-13408.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux af0e486e068e 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 28e2244 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24022/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24022/testReport/ |
| Max. process+thread count | 3146 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console outpu
[jira] [Commented] (HDFS-13488) RBF: Reject requests when a Router is overloaded
[
https://issues.apache.org/jira/browse/HDFS-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446554#comment-16446554
]
genericqa commented on HDFS-13488:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m
5s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13488 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920114/HDFS-13488.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 1b549ae7afb5 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 28e2244 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/24023/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24023/testReport/ |
| Max. process+thread count | 1328 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop
[jira] [Commented] (HDFS-13283) Percentage based Reserved Space Calculation for DataNode
[
https://issues.apache.org/jira/browse/HDFS-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446548#comment-16446548
]
genericqa commented on HDFS-13283:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-13283 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-13283 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920120/HDFS-13283.006.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24024/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Percentage based Reserved Space Calculation for DataNode
>
>
> Key: HDFS-13283
> URL: https://issues.apache.org/jira/browse/HDFS-13283
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Major
> Attachments: HDFS-13283.000.patch, HDFS-13283.001.patch,
> HDFS-13283.002.patch, HDFS-13283.003.patch, HDFS-13283.004.patch,
> HDFS-13283.005.patch, HDFS-13283.006.patch
>
>
> Currently, the only way to configure reserved disk space for non-HDFS data on
> a DataNode is a constant value via {{dfs.datanode.du.reserved}}. This can be
> an issue in non-heterogeneous clusters where size of DNs can differ. The
> proposed solution is to allow percentage based configuration (and their
> combination):
> # ABSOLUTE
> ** based on absolute number of reserved space
> # PERCENTAGE
> ** based on percentage of total capacity in the storage
> # CONSERVATIVE
> ** calculates both of the above and takes the one that will yield more
> reserved space
> # AGGRESSIVE
> ** calculates 1. 2. and takes the one that will yield less reserved space
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13283) Percentage based Reserved Space Calculation for DataNode
[
https://issues.apache.org/jira/browse/HDFS-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446545#comment-16446545
]
Lukas Majercak commented on HDFS-13283:
---
Sorry for being quiet on this. Thanks [~virajith] for the comments and
[~chris.douglas] for making the changes. I added patch006 to address the
comments for the documentation.
> Percentage based Reserved Space Calculation for DataNode
>
>
> Key: HDFS-13283
> URL: https://issues.apache.org/jira/browse/HDFS-13283
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Major
> Attachments: HDFS-13283.000.patch, HDFS-13283.001.patch,
> HDFS-13283.002.patch, HDFS-13283.003.patch, HDFS-13283.004.patch,
> HDFS-13283.005.patch, HDFS-13283.006.patch
>
>
> Currently, the only way to configure reserved disk space for non-HDFS data on
> a DataNode is a constant value via {{dfs.datanode.du.reserved}}. This can be
> an issue in non-heterogeneous clusters where size of DNs can differ. The
> proposed solution is to allow percentage based configuration (and their
> combination):
> # ABSOLUTE
> ** based on absolute number of reserved space
> # PERCENTAGE
> ** based on percentage of total capacity in the storage
> # CONSERVATIVE
> ** calculates both of the above and takes the one that will yield more
> reserved space
> # AGGRESSIVE
> ** calculates 1. 2. and takes the one that will yield less reserved space
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13283) Percentage based Reserved Space Calculation for DataNode
[
https://issues.apache.org/jira/browse/HDFS-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Majercak updated HDFS-13283:
--
Attachment: HDFS-13283.006.patch
> Percentage based Reserved Space Calculation for DataNode
>
>
> Key: HDFS-13283
> URL: https://issues.apache.org/jira/browse/HDFS-13283
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Major
> Attachments: HDFS-13283.000.patch, HDFS-13283.001.patch,
> HDFS-13283.002.patch, HDFS-13283.003.patch, HDFS-13283.004.patch,
> HDFS-13283.005.patch, HDFS-13283.006.patch
>
>
> Currently, the only way to configure reserved disk space for non-HDFS data on
> a DataNode is a constant value via {{dfs.datanode.du.reserved}}. This can be
> an issue in non-heterogeneous clusters where size of DNs can differ. The
> proposed solution is to allow percentage based configuration (and their
> combination):
> # ABSOLUTE
> ** based on absolute number of reserved space
> # PERCENTAGE
> ** based on percentage of total capacity in the storage
> # CONSERVATIVE
> ** calculates both of the above and takes the one that will yield more
> reserved space
> # AGGRESSIVE
> ** calculates 1. 2. and takes the one that will yield less reserved space
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13372) New Expunge Replica Trash Client-Namenode-Protocol
[ https://issues.apache.org/jira/browse/HDFS-13372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446537#comment-16446537 ] Ajay Kumar commented on HDFS-13372: --- [~bharatviswa], thanks for working on this. LGTM. 1 suggestion: * I think RouterRpcServer#expungeReplicaTrash should throw an exception with message indicating its not implemented. Not doing anything gives false impression of something working in background. > New Expunge Replica Trash Client-Namenode-Protocol > -- > > Key: HDFS-13372 > URL: https://issues.apache.org/jira/browse/HDFS-13372 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13372-HDFS-12996.00.patch > > > When client issues an expunge replica-trash RPC call to Namenode, the > Namenode will queue > a new heartbeat command response - DN_EXPUNGE directing the DataNodes to > expunge the > replica-trash. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10183) Prevent race condition during class initialization
[
https://issues.apache.org/jira/browse/HDFS-10183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446528#comment-16446528
]
Sangjin Lee commented on HDFS-10183:
Sorry it took a long time to accept this change. +1 from me. The test failures
are unrelated.
Which versions do we want to commit this to? Is trunk sufficient, or should
this go to another version too?
> Prevent race condition during class initialization
> --
>
> Key: HDFS-10183
> URL: https://issues.apache.org/jira/browse/HDFS-10183
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.9.0
>Reporter: Pavel Avgustinov
>Assignee: Pavel Avgustinov
>Priority: Minor
> Attachments: HADOOP-12944.1.patch, HDFS-10183.2.patch
>
>
> In HADOOP-11969, [~busbey] tracked down a non-deterministic
> {{NullPointerException}} to an oddity in the Java memory model: When multiple
> threads trigger the loading of a class at the same time, one of them wins and
> creates the {{java.lang.Class}} instance; the others block during this
> initialization, but once it is complete they may obtain a reference to the
> {{Class}} which has non-{{final}} fields still containing their default (i.e.
> {{null}}) values. This leads to runtime failures that are hard to debug or
> diagnose.
> HADOOP-11969 observed that {{ThreadLocal}} fields, by their very nature, are
> very likely to be accessed from multiple threads, and thus the problem is
> particularly severe there. Consequently, the patch removed all occurrences of
> the issue in the code base.
> Unfortunately, since then HDFS-7964 has [reverted one of the fixes during a
> refactoring|https://github.com/apache/hadoop/commit/2151716832ad14932dd65b1a4e47e64d8d6cd767#diff-0c2e9f7f9e685f38d1a11373b627cfa6R151],
> and introduced a [new instance of the
> problem|https://github.com/apache/hadoop/commit/2151716832ad14932dd65b1a4e47e64d8d6cd767#diff-6334d0df7d9aefbccd12b21bb7603169R43].
> The attached patch addresses the issue by adding the missing {{final}}
> modifier in these two cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13448) HDFS Block Placement - Ignore Locality for First Block Replica
[
https://issues.apache.org/jira/browse/HDFS-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446524#comment-16446524
]
Ajay Kumar commented on HDFS-13448:
---
[~belugabehr], [~daryn], [~kihwal] Rational behind suggesting config property
was to allow admin control over this. Many client run services like flume agent
on dedicated dense storage nodes. With flag its completely controlled by client
which may be outside HDFS (like flume agent). I don't feel strongly about this
but with a config, HDFS admins can still control such behavior if desired.
> HDFS Block Placement - Ignore Locality for First Block Replica
> --
>
> Key: HDFS-13448
> URL: https://issues.apache.org/jira/browse/HDFS-13448
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: block placement, hdfs-client
>Affects Versions: 2.9.0, 3.0.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-13448.1.patch, HDFS-13448.2.patch,
> HDFS-13448.3.patch, HDFS-13448.4.patch, HDFS-13448.5.patch
>
>
> According to the HDFS Block Place Rules:
> {quote}
> /**
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on a different node of the rack as the second replica.
> */
> {quote}
> However, there is a hint for the hdfs-client that allows the block placement
> request to not put a block replica on the local datanode _where 'local' means
> the same host as the client is being run on._
> {quote}
> /**
>* Advise that a block replica NOT be written to the local DataNode where
>* 'local' means the same host as the client is being run on.
>*
>* @see CreateFlag#NO_LOCAL_WRITE
>*/
> {quote}
> I propose that we add a new flag that allows the hdfs-client to request that
> the first block replica be placed on a random DataNode in the cluster. The
> subsequent block replicas should follow the normal block placement rules.
> The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block
> replica is not placed on the local node, but it is still placed on the local
> rack. Where this comes into play is where you have, for example, a flume
> agent that is loading data into HDFS.
> If the Flume agent is running on a DataNode, then by default, the DataNode
> local to the Flume agent will always get the first block replica and this
> leads to un-even block placements, with the local node always filling up
> faster than any other node in the cluster.
> Modifying this example, if the DataNode is removed from the host where the
> Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then
> the default block placement policy will still prefer the local rack. This
> remedies the situation only so far as now the first block replica will always
> be distributed to a DataNode on the local rack.
> This new flag would allow a single Flume agent to distribute the blocks
> randomly, evenly, over the entire cluster instead of hot-spotting the local
> node or the local rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13399) Make Client field AlignmentContext non-static.
[
https://issues.apache.org/jira/browse/HDFS-13399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446514#comment-16446514
]
Konstantin Shvachko commented on HDFS-13399:
# It seems to me that {{createNonHAProxy()}} does not generally need
{{AlignmentContext}}, because without HA there are no Standbys and therefore no
Observers. Should we remove those changes?
# I think it is incorrect to add {{AlignmentContext}} to
{{AbstractNNFailoverProxyProvider}}, which is inherited by all ProxyProviders.
Some ProxyProvider are meant to talk to single NN only, so the
{{AlignmentContext}} should be present exclusively in
{{StandbyReadProxyProvider}} for now. If you need this for testing, then you
should be able to define in the tests some
{{TestProxyProviderWithAlignmentContext}} derived from whatever variant of
ProxyProvider you need, and use it there.
> Make Client field AlignmentContext non-static.
> --
>
> Key: HDFS-13399
> URL: https://issues.apache.org/jira/browse/HDFS-13399
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13399-HDFS-12943.000.patch,
> HDFS-13399-HDFS-12943.001.patch, HDFS-13399-HDFS-12943.002.patch
>
>
> In HDFS-12977, DFSClient's constructor was altered to make use of a new
> static method in Client that allowed one to set an AlignmentContext. This
> work is to remove that static field and make each DFSClient pass it's
> AlignmentContext down to the proxy Call level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13488) RBF: Reject requests when a Router is overloaded
[
https://issues.apache.org/jira/browse/HDFS-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446511#comment-16446511
]
Íñigo Goiri commented on HDFS-13488:
After adding the unit test for the client failover across Routers, I realized
that the exception has to be {{StandbyException}} for it to work.
It cannot just be a subclass as the retry policy uses
{{RemoteException#unwrapRemoteException(StandbyException.class)}}.
For this reason I removed the {{RouterSafeModeException}}.
Other than that, [^HDFS-13488.000.patch] should be able to the resiliency to
overloaded subclusters.
To handle these scenarios, one can also use HDFS-13484.
> RBF: Reject requests when a Router is overloaded
>
>
> Key: HDFS-13488
> URL: https://issues.apache.org/jira/browse/HDFS-13488
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-13488.000.patch
>
>
> A Router might be overloaded when handling special cases (e.g. a slow
> subcluster). The Router could reject the requests and the client could try
> with another Router. We should leverage the Standby mechanism for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13488) RBF: Reject requests when a Router is overloaded
[ https://issues.apache.org/jira/browse/HDFS-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13488: --- Status: Patch Available (was: Open) > RBF: Reject requests when a Router is overloaded > > > Key: HDFS-13488 > URL: https://issues.apache.org/jira/browse/HDFS-13488 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13488.000.patch > > > A Router might be overloaded when handling special cases (e.g. a slow > subcluster). The Router could reject the requests and the client could try > with another Router. We should leverage the Standby mechanism for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13488) RBF: Reject requests when a Router is overloaded
[ https://issues.apache.org/jira/browse/HDFS-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13488: --- Attachment: HDFS-13488.000.patch > RBF: Reject requests when a Router is overloaded > > > Key: HDFS-13488 > URL: https://issues.apache.org/jira/browse/HDFS-13488 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13488.000.patch > > > A Router might be overloaded when handling special cases (e.g. a slow > subcluster). The Router could reject the requests and the client could try > with another Router. We should leverage the Standby mechanism for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10867) [PROVIDED Phase 2] Block Bit Field Allocation of Provided Storage
[ https://issues.apache.org/jira/browse/HDFS-10867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446483#comment-16446483 ] Chris Douglas commented on HDFS-10867: -- Looking through the patch for HDFS-13350, it looks like it's handling false positives where legacy blocks were treated as EC blocks, not the other way around. If we're already identifying legacy blocks correctly, then we need to avoid the same pitfall, but the rest of the code should be OK. > [PROVIDED Phase 2] Block Bit Field Allocation of Provided Storage > - > > Key: HDFS-10867 > URL: https://issues.apache.org/jira/browse/HDFS-10867 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Ewan Higgs >Priority: Major > Attachments: Block Bit Field Allocation of Provided Storage.pdf > > > We wish to design and implement the following related features for provided > storage: > # Dynamic mounting of provided storage within a Namenode (mount, unmount) > # Mount multiple provided storage systems on a single Namenode. > # Support updates to the provided storage system without having to regenerate > an fsimg. > A mount in the namespace addresses a corresponding set of block data. When > unmounted, any block data associated with the mount becomes invalid and > (eventually) unaddressable in HDFS. As with erasure-coded blocks, efficient > unmounting requires that all blocks with that attribute be identifiable by > the block management layer > In this subtask, we focus on changes and conventions to the block management > layer. Namespace operations are covered in a separate subtask. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13408) MiniDFSCluster to support being built on randomized base directory
[ https://issues.apache.org/jira/browse/HDFS-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446478#comment-16446478 ] Íñigo Goiri commented on HDFS-13408: +1 on [^HDFS-13408.004.patch]. Let's see if the unit tests pass. > MiniDFSCluster to support being built on randomized base directory > -- > > Key: HDFS-13408 > URL: https://issues.apache.org/jira/browse/HDFS-13408 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Xiao Liang >Assignee: Xiao Liang >Priority: Major > Labels: windows > Attachments: HDFS-13408.000.patch, HDFS-13408.001.patch, > HDFS-13408.002.patch, HDFS-13408.003.patch, HDFS-13408.004.patch > > > Generated files of MiniDFSCluster during test are not properly cleaned in > Windows, which fails all subsequent test cases using the same default > directory (Windows does not allow other processes to delete them). By > migrating to randomized base directories, the conflict of test path of test > cases will be avoided, even if they are running at the same time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13272) DataNodeHttpServer to have configurable HttpServer2 threads
[
https://issues.apache.org/jira/browse/HDFS-13272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446469#comment-16446469
]
Chris Douglas commented on HDFS-13272:
--
Thanks, [~kihwal]. [~xkrogen], does this unblock you on HDFS-13265?
> DataNodeHttpServer to have configurable HttpServer2 threads
> ---
>
> Key: HDFS-13272
> URL: https://issues.apache.org/jira/browse/HDFS-13272
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
>
> In HDFS-7279, the Jetty server on the DataNode was hard-coded to use 10
> threads. In addition to the possibility of this being too few threads, it is
> much higher than necessary in resource constrained environments such as
> MiniDFSCluster. To avoid compatibility issues, rather than using
> {{HttpServer2#HTTP_MAX_THREADS}} directly, we can introduce a new
> configuration for the DataNode's thread pool size.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13408) MiniDFSCluster to support being built on randomized base directory
[
https://issues.apache.org/jira/browse/HDFS-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446457#comment-16446457
]
Chris Douglas commented on HDFS-13408:
--
For completeness, added v004 which considers a {{null}} base directory to the
new cstr an error.
> MiniDFSCluster to support being built on randomized base directory
> --
>
> Key: HDFS-13408
> URL: https://issues.apache.org/jira/browse/HDFS-13408
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Xiao Liang
>Assignee: Xiao Liang
>Priority: Major
> Labels: windows
> Attachments: HDFS-13408.000.patch, HDFS-13408.001.patch,
> HDFS-13408.002.patch, HDFS-13408.003.patch, HDFS-13408.004.patch
>
>
> Generated files of MiniDFSCluster during test are not properly cleaned in
> Windows, which fails all subsequent test cases using the same default
> directory (Windows does not allow other processes to delete them). By
> migrating to randomized base directories, the conflict of test path of test
> cases will be avoided, even if they are running at the same time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13408) MiniDFSCluster to support being built on randomized base directory
[ https://issues.apache.org/jira/browse/HDFS-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HDFS-13408: - Attachment: HDFS-13408.004.patch > MiniDFSCluster to support being built on randomized base directory > -- > > Key: HDFS-13408 > URL: https://issues.apache.org/jira/browse/HDFS-13408 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Xiao Liang >Assignee: Xiao Liang >Priority: Major > Labels: windows > Attachments: HDFS-13408.000.patch, HDFS-13408.001.patch, > HDFS-13408.002.patch, HDFS-13408.003.patch, HDFS-13408.004.patch > > > Generated files of MiniDFSCluster during test are not properly cleaned in > Windows, which fails all subsequent test cases using the same default > directory (Windows does not allow other processes to delete them). By > migrating to randomized base directories, the conflict of test path of test > cases will be avoided, even if they are running at the same time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13408) MiniDFSCluster to support being built on randomized base directory
[ https://issues.apache.org/jira/browse/HDFS-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446454#comment-16446454 ] Chris Douglas commented on HDFS-13408: -- bq. there should be file read/write conflicts between test cases using the same base dir and test data file names, if running in parallel. OK. Perhaps we could [use the ID from surefire|https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html] to avoid these collisions. We still need a solution either for tests run sequentially, and this is a good workaround. bq. Regarding the warning for setting both approaches, I'm fine with it and I would even vote for an exception to avoid having inconsistencies Soright. Updated patch, which also rolls back the slf4j change. > MiniDFSCluster to support being built on randomized base directory > -- > > Key: HDFS-13408 > URL: https://issues.apache.org/jira/browse/HDFS-13408 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Xiao Liang >Assignee: Xiao Liang >Priority: Major > Labels: windows > Attachments: HDFS-13408.000.patch, HDFS-13408.001.patch, > HDFS-13408.002.patch, HDFS-13408.003.patch > > > Generated files of MiniDFSCluster during test are not properly cleaned in > Windows, which fails all subsequent test cases using the same default > directory (Windows does not allow other processes to delete them). By > migrating to randomized base directories, the conflict of test path of test > cases will be avoided, even if they are running at the same time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13408) MiniDFSCluster to support being built on randomized base directory
[ https://issues.apache.org/jira/browse/HDFS-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HDFS-13408: - Attachment: HDFS-13408.003.patch > MiniDFSCluster to support being built on randomized base directory > -- > > Key: HDFS-13408 > URL: https://issues.apache.org/jira/browse/HDFS-13408 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Xiao Liang >Assignee: Xiao Liang >Priority: Major > Labels: windows > Attachments: HDFS-13408.000.patch, HDFS-13408.001.patch, > HDFS-13408.002.patch, HDFS-13408.003.patch > > > Generated files of MiniDFSCluster during test are not properly cleaned in > Windows, which fails all subsequent test cases using the same default > directory (Windows does not allow other processes to delete them). By > migrating to randomized base directories, the conflict of test path of test > cases will be avoided, even if they are running at the same time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10943) rollEditLog expects empty EditsDoubleBuffer.bufCurrent which is not guaranteed
[
https://issues.apache.org/jira/browse/HDFS-10943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446446#comment-16446446
]
Daryn Sharp commented on HDFS-10943:
The quorum output stream will +always+ crash the NN if edits are rolled while a
node is down. The edit roll will sync, flush, close. As we can see, closing a
logger will fail if there are unflushed bytes.
The issue is {{QuorumOutputStream#flush}} succeeds with only a simple majority
of loggers. The those that failed cause close to fail due to unflushed bytes.
Someone familiar with QJM (we don't use it) will need to decide if it's safe
for the quorum's flush to clear the buffers of failed loggers.
> rollEditLog expects empty EditsDoubleBuffer.bufCurrent which is not guaranteed
> --
>
> Key: HDFS-10943
> URL: https://issues.apache.org/jira/browse/HDFS-10943
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Priority: Major
>
> Per the following trace stack:
> {code}
> FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: finalize log
> segment 10562075963, 10562174157 failed for required journal
> (JournalAndStream(mgr=QJM to [0.0.0.1:8485, 0.0.0.2:8485, 0.0.0.3:8485,
> 0.0.0.4:8485, 0.0.0.5:8485], stream=QuorumOutputStream starting at txid
> 10562075963))
> java.io.IOException: FSEditStream has 49708 bytes still to be flushed and
> cannot be closed.
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer.close(EditsDoubleBuffer.java:66)
> at
> org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.close(QuorumOutputStream.java:65)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalAndStream.closeStream(JournalSet.java:115)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$4.apply(JournalSet.java:235)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:231)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:1243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:6437)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:1002)
> at
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:142)
> at
> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:12025)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
> 2016-09-23 21:40:59,618 WARN
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Aborting
> QuorumOutputStream starting at txid 10562075963
> {code}
> The exception is from EditsDoubleBuffer
> {code}
> public void close() throws IOException {
> Preconditions.checkNotNull(bufCurrent);
> Preconditions.checkNotNull(bufReady);
> int bufSize = bufCurrent.size();
> if (bufSize != 0) {
> throw new IOException("FSEditStream has " + bufSize
> + " bytes still to be flushed and cannot be closed.");
> }
> IOUtils.cleanup(null, bufCurrent, bufReady);
> bufCurrent = bufReady = null;
> }
> {code}
> We can see that FSNamesystem.rollEditLog expects
> EditsDoubleBuffer.bufCurrent to be empty.
> Edits are recorded via FSEditLog$logSync, which does:
> {code}
>* The data is double-buffered within each edit log implementation so that
>* in-memory writing can occur in parallel with the on-disk writing.
>*
>* Each sync occurs in three steps:
>* 1. synchronized, it swaps the double buffer and sets the isSyncRunning
>* flag.
>* 2. unsynchronized, it flushes the data to storage
>* 3. synchronized, it resets the flag and notifies anyone waiting on the
>* sync.
>*
>*
[jira] [Updated] (HDFS-13468) Add erasure coding metrics into ReadStatistics
[ https://issues.apache.org/jira/browse/HDFS-13468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-13468: - Attachment: HDFS-13468.01.patch > Add erasure coding metrics into ReadStatistics > -- > > Key: HDFS-13468 > URL: https://issues.apache.org/jira/browse/HDFS-13468 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.1.0, 3.0.1 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu >Priority: Major > Attachments: HDFS-13468.00.patch, HDFS-13468.01.patch > > > Expose Erasure Coding related metrics for InputStream in ReadStatistics. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13468) Add erasure coding metrics into ReadStatistics
[ https://issues.apache.org/jira/browse/HDFS-13468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446445#comment-16446445 ] Lei (Eddy) Xu commented on HDFS-13468: -- Update patch to fix checkstyle warnings. > Add erasure coding metrics into ReadStatistics > -- > > Key: HDFS-13468 > URL: https://issues.apache.org/jira/browse/HDFS-13468 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.1.0, 3.0.1 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu >Priority: Major > Attachments: HDFS-13468.00.patch, HDFS-13468.01.patch > > > Expose Erasure Coding related metrics for InputStream in ReadStatistics. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13468) Add erasure coding metrics into ReadStatistics
[
https://issues.apache.org/jira/browse/HDFS-13468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446437#comment-16446437
]
genericqa commented on HDFS-13468:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 57s{color} | {color:orange} hadoop-hdfs-project: The patch generated 5 new +
93 unchanged - 0 fixed = 98 total (was 93) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
24s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 76m
6s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}177m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13468 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920067/HDFS-13468.00.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 883687ca7160 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 766544c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/24020/artifact/
[jira] [Commented] (HDFS-10183) Prevent race condition during class initialization
[
https://issues.apache.org/jira/browse/HDFS-10183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446375#comment-16446375
]
genericqa commented on HDFS-10183:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 54s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 45s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-10183 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12794395/HDFS-10183.2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 009be1e12d47 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 766544c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24019/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCo
[jira] [Comment Edited] (HDFS-13373) Handle expunge command on NN and DN
[
https://issues.apache.org/jira/browse/HDFS-13373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446358#comment-16446358
]
Hanisha Koneru edited comment on HDFS-13373 at 4/20/18 9:00 PM:
Thanks for the patch [~bharatviswa].
# In {{PBHelper#convert(ReplicaTrashCommandProto)}}, the switch case should
have a default case which throws AssertionError (please refer to
{{BlockCommandProto convert(BlockCommand cmd)}}).
# In {{BPOfferService#processCommandFromActive}}, can we move the check whether
the command is ReplicaTrashCommand inside the switch case so as to avoid making
this check for every Datanode command.
{code}
final ReplicaTrashCommand rcmd = cmd instanceof ReplicaTrashCommand ?
(ReplicaTrashCommand)cmd : null;
{code}
# We should handle the case where blocks are deleted and moved to Replica Trash
after the expunge command is issued. These new blocks should not be removed
from the Replica Trash. One option is to note down the timestamp when expunge
command was received. All invalidated blocks moved to replica trash after this
timestamp will not be expunged.
was (Author: hanishakoneru):
Thanks for the patch [~bharatviswa].
# In {{PBHelper#convert(ReplicaTrashCommandProto)}}, the switch case should
have a default case which throws AssertionError (please refer to
{{BlockCommandProto convert(BlockCommand cmd)}}).
# In {{BPOfferService#processCommandFromActive}}, can we move the check whether
the command is ReplicaTrashCommand inside the switch case so as to avoid making
this check for every Datanode command.
{code}
final ReplicaTrashCommand rcmd = cmd instanceof ReplicaTrashCommand ?
(ReplicaTrashCommand)cmd : null;
{code}
# We should handle the case where blocks are deleted and moved to Replica Trash
after the expunge command is issued. These new blocks should not be removed
from the Replica Trash. One option is to note down the timestamp when expunge
command was received. All invalidated blocks moved to replica trash after this
timestamp should not be expunged.
> Handle expunge command on NN and DN
> ---
>
> Key: HDFS-13373
> URL: https://issues.apache.org/jira/browse/HDFS-13373
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13373-HDFS-12996.00.patch
>
>
> When DataNodes receive the DN_EXPUNGE command from Namenode, they will
> purge all the block replicas in replica-trash
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13373) Handle expunge command on NN and DN
[
https://issues.apache.org/jira/browse/HDFS-13373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446358#comment-16446358
]
Hanisha Koneru commented on HDFS-13373:
---
Thanks for the patch [~bharatviswa].
# In {{PBHelper#convert(ReplicaTrashCommandProto)}}, the switch case should
have a default case which throws AssertionError (please refer to
{{BlockCommandProto convert(BlockCommand cmd)}}).
# In {{BPOfferService#processCommandFromActive}}, can we move the check whether
the command is ReplicaTrashCommand inside the switch case so as to avoid making
this check for every Datanode command.
{code}
final ReplicaTrashCommand rcmd = cmd instanceof ReplicaTrashCommand ?
(ReplicaTrashCommand)cmd : null;
{code}
# We should handle the case where blocks are deleted and moved to Replica Trash
after the expunge command is issued. These new blocks should not be removed
from the Replica Trash. One option is to note down the timestamp when expunge
command was received. All invalidated blocks moved to replica trash after this
timestamp should not be expunged.
> Handle expunge command on NN and DN
> ---
>
> Key: HDFS-13373
> URL: https://issues.apache.org/jira/browse/HDFS-13373
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13373-HDFS-12996.00.patch
>
>
> When DataNodes receive the DN_EXPUNGE command from Namenode, they will
> purge all the block replicas in replica-trash
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10943) rollEditLog expects empty EditsDoubleBuffer.bufCurrent which is not guaranteed
[
https://issues.apache.org/jira/browse/HDFS-10943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446329#comment-16446329
]
Daryn Sharp commented on HDFS-10943:
This error should not be possible. The entire roll holds the FSN lock so
nothing else can generate edits to avoid lingering bytes in the buffer.
{{endCurrentLogSegment}} does a logSync which loops calling setReadyToFlush
(which flips the buffer) & flush until the last written txid is synced. If the
stream reports the last txid synced, there cannot/must-not be bytes lingering
in the buffer or the stream lied.
I'll take a quick look at the code.
> rollEditLog expects empty EditsDoubleBuffer.bufCurrent which is not guaranteed
> --
>
> Key: HDFS-10943
> URL: https://issues.apache.org/jira/browse/HDFS-10943
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Priority: Major
>
> Per the following trace stack:
> {code}
> FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: finalize log
> segment 10562075963, 10562174157 failed for required journal
> (JournalAndStream(mgr=QJM to [0.0.0.1:8485, 0.0.0.2:8485, 0.0.0.3:8485,
> 0.0.0.4:8485, 0.0.0.5:8485], stream=QuorumOutputStream starting at txid
> 10562075963))
> java.io.IOException: FSEditStream has 49708 bytes still to be flushed and
> cannot be closed.
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer.close(EditsDoubleBuffer.java:66)
> at
> org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.close(QuorumOutputStream.java:65)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalAndStream.closeStream(JournalSet.java:115)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$4.apply(JournalSet.java:235)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:231)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:1243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:6437)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:1002)
> at
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:142)
> at
> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:12025)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
> 2016-09-23 21:40:59,618 WARN
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Aborting
> QuorumOutputStream starting at txid 10562075963
> {code}
> The exception is from EditsDoubleBuffer
> {code}
> public void close() throws IOException {
> Preconditions.checkNotNull(bufCurrent);
> Preconditions.checkNotNull(bufReady);
> int bufSize = bufCurrent.size();
> if (bufSize != 0) {
> throw new IOException("FSEditStream has " + bufSize
> + " bytes still to be flushed and cannot be closed.");
> }
> IOUtils.cleanup(null, bufCurrent, bufReady);
> bufCurrent = bufReady = null;
> }
> {code}
> We can see that FSNamesystem.rollEditLog expects
> EditsDoubleBuffer.bufCurrent to be empty.
> Edits are recorded via FSEditLog$logSync, which does:
> {code}
>* The data is double-buffered within each edit log implementation so that
>* in-memory writing can occur in parallel with the on-disk writing.
>*
>* Each sync occurs in three steps:
>* 1. synchronized, it swaps the double buffer and sets the isSyncRunning
>* flag.
>* 2. unsynchronized, it flushes the data to storage
>* 3. synchronized, it resets the flag and notifies anyone waiting on the
>* sync.
>*
>* The lack of synchronization on step 2 allows other thr
[jira] [Commented] (HDFS-13372) New Expunge Replica Trash Client-Namenode-Protocol
[
https://issues.apache.org/jira/browse/HDFS-13372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446270#comment-16446270
]
Hanisha Koneru commented on HDFS-13372:
---
Thanks for working on this [~bharatviswa]. The patch LGTM overall.
Just two NITs:
# In {{DFSAdmin#expungeReplicaTrash} there is a typo in the System.out message
-> "operation is queued and will be sent to -in- datanodes".
# In {{RouterRpcServer}}, can you please remove the space between {{TO DO}} and
add a description on what the todo is. Just so that it shows up when viewing
TODO items in editor tool window.
Will trigger Jenkins manually if it doesn't run next time as well.
> New Expunge Replica Trash Client-Namenode-Protocol
> --
>
> Key: HDFS-13372
> URL: https://issues.apache.org/jira/browse/HDFS-13372
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13372-HDFS-12996.00.patch
>
>
> When client issues an expunge replica-trash RPC call to Namenode, the
> Namenode will queue
> a new heartbeat command response - DN_EXPUNGE directing the DataNodes to
> expunge the
> replica-trash.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13468) Add erasure coding metrics into ReadStatistics
[ https://issues.apache.org/jira/browse/HDFS-13468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-13468: - Status: Patch Available (was: Open) > Add erasure coding metrics into ReadStatistics > -- > > Key: HDFS-13468 > URL: https://issues.apache.org/jira/browse/HDFS-13468 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.1, 3.1.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu >Priority: Major > Attachments: HDFS-13468.00.patch > > > Expose Erasure Coding related metrics for InputStream in ReadStatistics. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13468) Add erasure coding metrics into ReadStatistics
[ https://issues.apache.org/jira/browse/HDFS-13468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-13468: - Attachment: HDFS-13468.00.patch > Add erasure coding metrics into ReadStatistics > -- > > Key: HDFS-13468 > URL: https://issues.apache.org/jira/browse/HDFS-13468 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.1.0, 3.0.1 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu >Priority: Major > Attachments: HDFS-13468.00.patch > > > Expose Erasure Coding related metrics for InputStream in ReadStatistics. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13485) DataNode WebHDFS endpoint throws NPE
[
https://issues.apache.org/jira/browse/HDFS-13485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446062#comment-16446062
]
Wei-Chiu Chuang commented on HDFS-13485:
Thank you [~kihwal] yes I should have updated the jira summary.
That said it would be nice to have the parameters checked properly
> DataNode WebHDFS endpoint throws NPE
>
>
> Key: HDFS-13485
> URL: https://issues.apache.org/jira/browse/HDFS-13485
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, webhdfs
>Affects Versions: 3.0.0
> Environment: Kerberized. Hadoop 3.0.0, WebHDFS.
>Reporter: Wei-Chiu Chuang
>Priority: Minor
>
> curl -k -i --negotiate -u : "https://hadoop3-4.example.com:20004/webhdfs/v1";
> DataNode Web UI should do a better error checking/handling.
> {noformat}
> 2018-04-19 10:07:49,338 WARN
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler:
> INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.token.Token.decodeWritable(Token.java:364)
> at
> org.apache.hadoop.security.token.Token.decodeFromUrlString(Token.java:383)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.ParameterParser.delegationToken(ParameterParser.java:128)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.DataNodeUGIProvider.ugi(DataNodeUGIProvider.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:129)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:51)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:31)
> at
> com.cloudera.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1379)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1158)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1193)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead
[jira] [Updated] (HDFS-13485) DataNode WebHDFS endpoint throws NPE
[
https://issues.apache.org/jira/browse/HDFS-13485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-13485:
---
Summary: DataNode WebHDFS endpoint throws NPE (was: DataNode WebUI throws
NPE)
> DataNode WebHDFS endpoint throws NPE
>
>
> Key: HDFS-13485
> URL: https://issues.apache.org/jira/browse/HDFS-13485
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, webhdfs
>Affects Versions: 3.0.0
> Environment: Kerberized. Hadoop 3.0.0, WebHDFS.
>Reporter: Wei-Chiu Chuang
>Priority: Minor
>
> curl -k -i --negotiate -u : "https://hadoop3-4.example.com:20004/webhdfs/v1";
> DataNode Web UI should do a better error checking/handling.
> {noformat}
> 2018-04-19 10:07:49,338 WARN
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler:
> INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.token.Token.decodeWritable(Token.java:364)
> at
> org.apache.hadoop.security.token.Token.decodeFromUrlString(Token.java:383)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.ParameterParser.delegationToken(ParameterParser.java:128)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.DataNodeUGIProvider.ugi(DataNodeUGIProvider.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.web.webhdfs.WebHdfsHandler.channelRead0(WebHdfsHandler.java:129)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:51)
> at
> org.apache.hadoop.hdfs.server.datanode.web.URLDispatcher.channelRead0(URLDispatcher.java:31)
> at
> com.cloudera.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1379)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1158)
> at
> com.cloudera.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1193)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
> at
> com.cloudera.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> com.cloudera.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> com.cloudera.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> com.cloudera.io.netty.channel.DefaultChannelPip
[jira] [Commented] (HDFS-13403) libhdfs++: Use hdfs::IoService object rather than asio::io_service
[ https://issues.apache.org/jira/browse/HDFS-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446056#comment-16446056 ] Istvan Fajth commented on HDFS-13403: - Hi [~James C], yes, those are one type of the warnings, also I see warnings due to -Wunused-lambda-capture though after checking a few instance of this warning, they not seems to be related to this patch, though I just vaguely checked, but most likely they were there before as well. With the patch applied, I do not see inconsistent-missing-override warnings, and the include helped as well to get over the compile error. > libhdfs++: Use hdfs::IoService object rather than asio::io_service > -- > > Key: HDFS-13403 > URL: https://issues.apache.org/jira/browse/HDFS-13403 > Project: Hadoop HDFS > Issue Type: Task >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-13403.000.patch, build_fixes.patch > > > At the moment the hdfs::IoService is a simple wrapper over asio's io_service > object. I'd like to make this smarter and have it do things like track which > tasks are queued, validate that dependencies of tasks exist, and monitor > ioservice throughput and contention. In order to get there we need to use > have all components in the library to go through the hdfs::IoService rather > than directly interacting with the asio::io_service. The only time the > asio::io_service should be used is when calling things like asio::async_write > that need an io_service&. HDFS-11884 will be able get rid of those remaining > instances once this work is in place. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13478) RBF: Disabled Nameservice store API
[
https://issues.apache.org/jira/browse/HDFS-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446052#comment-16446052
]
genericqa commented on HDFS-13478:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
32s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13478 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920036/HDFS-13478.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 67acca897ff0 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fd24fd0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24018/testReport/ |
| Max. process+thread count | 988 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24018/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> RBF: Disabled Namese
[jira] [Comment Edited] (HDFS-13489) Get base snapshotable path if exists for a given path
[ https://issues.apache.org/jira/browse/HDFS-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446003#comment-16446003 ] Gabor Bota edited comment on HDFS-13489 at 4/20/18 4:43 PM: Of course, I just wanted to watch the issue, but clicked on the wrong link. Sorry for that. was (Author: gabor.bota): Of course, I just wanted to watch the issue, but clicked on thee wrong link. Sorry for that. > Get base snapshotable path if exists for a given path > - > > Key: HDFS-13489 > URL: https://issues.apache.org/jira/browse/HDFS-13489 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs >Reporter: Harkrishn Patro >Assignee: Harkrishn Patro >Priority: Major > > Currently, hdfs only lists the snapshotable paths in the filesystem. This > feature would add the functionality of figuring out if a given path is > snapshotable or not. If yes, it would return the base snapshotable path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13489) Get base snapshotable path if exists for a given path
[ https://issues.apache.org/jira/browse/HDFS-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446003#comment-16446003 ] Gabor Bota commented on HDFS-13489: --- Of course, I just wanted to watch the issue, but clicked on thee wrong link. Sorry for that. > Get base snapshotable path if exists for a given path > - > > Key: HDFS-13489 > URL: https://issues.apache.org/jira/browse/HDFS-13489 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs >Reporter: Harkrishn Patro >Assignee: Harkrishn Patro >Priority: Major > > Currently, hdfs only lists the snapshotable paths in the filesystem. This > feature would add the functionality of figuring out if a given path is > snapshotable or not. If yes, it would return the base snapshotable path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13489) Get base snapshotable path if exists for a given path
[ https://issues.apache.org/jira/browse/HDFS-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harkrishn Patro reassigned HDFS-13489: -- Assignee: Harkrishn Patro (was: Gabor Bota) > Get base snapshotable path if exists for a given path > - > > Key: HDFS-13489 > URL: https://issues.apache.org/jira/browse/HDFS-13489 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs >Reporter: Harkrishn Patro >Assignee: Harkrishn Patro >Priority: Major > > Currently, hdfs only lists the snapshotable paths in the filesystem. This > feature would add the functionality of figuring out if a given path is > snapshotable or not. If yes, it would return the base snapshotable path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13489) Get base snapshotable path if exists for a given path
[ https://issues.apache.org/jira/browse/HDFS-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445995#comment-16445995 ] Harkrishn Patro commented on HDFS-13489: Yes, [~jojochuang]. Hey [~gabor.bota], I have already started working on this and have a patch available. So picking it up. Hope this is fine with you. > Get base snapshotable path if exists for a given path > - > > Key: HDFS-13489 > URL: https://issues.apache.org/jira/browse/HDFS-13489 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs >Reporter: Harkrishn Patro >Assignee: Gabor Bota >Priority: Major > > Currently, hdfs only lists the snapshotable paths in the filesystem. This > feature would add the functionality of figuring out if a given path is > snapshotable or not. If yes, it would return the base snapshotable path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13408) MiniDFSCluster to support being built on randomized base directory
[ https://issues.apache.org/jira/browse/HDFS-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445972#comment-16445972 ] Íñigo Goiri commented on HDFS-13408: The Yetus run for [^HDFS-13408.002.patch] seems good; the unit test are unrelated. Regarding the warning for setting both approaches, I'm fine with it and I would even vote for an exception to avoid having inconsistencies. > MiniDFSCluster to support being built on randomized base directory > -- > > Key: HDFS-13408 > URL: https://issues.apache.org/jira/browse/HDFS-13408 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Xiao Liang >Assignee: Xiao Liang >Priority: Major > Labels: windows > Attachments: HDFS-13408.000.patch, HDFS-13408.001.patch, > HDFS-13408.002.patch > > > Generated files of MiniDFSCluster during test are not properly cleaned in > Windows, which fails all subsequent test cases using the same default > directory (Windows does not allow other processes to delete them). By > migrating to randomized base directories, the conflict of test path of test > cases will be avoided, even if they are running at the same time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13489) Get base snapshotable path if exists for a given path
[
https://issues.apache.org/jira/browse/HDFS-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445980#comment-16445980
]
Wei-Chiu Chuang commented on HDFS-13489:
Thanks for filing the issue. You should be able to use
{{SnapshotManager#getSnapshottableRoot()}} for the purpose. But I guess you
want to expose a client visible API?
> Get base snapshotable path if exists for a given path
> -
>
> Key: HDFS-13489
> URL: https://issues.apache.org/jira/browse/HDFS-13489
> Project: Hadoop HDFS
> Issue Type: Task
> Components: hdfs
>Reporter: Harkrishn Patro
>Assignee: Gabor Bota
>Priority: Major
>
> Currently, hdfs only lists the snapshotable paths in the filesystem. This
> feature would add the functionality of figuring out if a given path is
> snapshotable or not. If yes, it would return the base snapshotable path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13245) RBF: State store DBMS implementation
[ https://issues.apache.org/jira/browse/HDFS-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445964#comment-16445964 ] Íñigo Goiri commented on HDFS-13245: A couple comments: * Should the sql scripts go into the RBF module folder itself? * Is there any easy way to get a unit test? In-memory SQL server? Mocking? > RBF: State store DBMS implementation > > > Key: HDFS-13245 > URL: https://issues.apache.org/jira/browse/HDFS-13245 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: maobaolong >Assignee: Yiran Wu >Priority: Major > Attachments: HDFS-13245.001.patch, HDFS-13245.002.patch > > > Add a DBMS implementation for the State Store. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13478) RBF: Disabled Nameservice store API
[ https://issues.apache.org/jira/browse/HDFS-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445951#comment-16445951 ] Íñigo Goiri commented on HDFS-13478: Thanks [~linyiqun] for the comments, tackled them in [^HDFS-13478.005.patch]. I also had some personal debate on the terminology and finally decided to make everything consistent into disable/enable terminology instead of decommission. Right now, everything should be consistent; I updated the JIRA name and HDFS-13484 will follow this terminology too. > RBF: Disabled Nameservice store API > --- > > Key: HDFS-13478 > URL: https://issues.apache.org/jira/browse/HDFS-13478 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13478.000.patch, HDFS-13478.001.patch, > HDFS-13478.002.patch, HDFS-13478.003.patch, HDFS-13478.004.patch, > HDFS-13478.005.patch > > > We have a subcluster in our federation that is for testing and is > missbehaving. This has a negative impact on the performance with operations > that go to every subcluster (e.g., renewLease() or setSafeMode()). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13478) RBF: Disabled Nameservice store API
[ https://issues.apache.org/jira/browse/HDFS-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13478: --- Attachment: HDFS-13478.005.patch > RBF: Disabled Nameservice store API > --- > > Key: HDFS-13478 > URL: https://issues.apache.org/jira/browse/HDFS-13478 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13478.000.patch, HDFS-13478.001.patch, > HDFS-13478.002.patch, HDFS-13478.003.patch, HDFS-13478.004.patch, > HDFS-13478.005.patch > > > We have a subcluster in our federation that is for testing and is > missbehaving. This has a negative impact on the performance with operations > that go to every subcluster (e.g., renewLease() or setSafeMode()). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13489) Get base snapshotable path if exists for a given path
[ https://issues.apache.org/jira/browse/HDFS-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HDFS-13489: - Assignee: Gabor Bota > Get base snapshotable path if exists for a given path > - > > Key: HDFS-13489 > URL: https://issues.apache.org/jira/browse/HDFS-13489 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs >Reporter: Harkrishn Patro >Assignee: Gabor Bota >Priority: Major > > Currently, hdfs only lists the snapshotable paths in the filesystem. This > feature would add the functionality of figuring out if a given path is > snapshotable or not. If yes, it would return the base snapshotable path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13489) Get base snapshotable path if exists for a given path
Harkrishn Patro created HDFS-13489: -- Summary: Get base snapshotable path if exists for a given path Key: HDFS-13489 URL: https://issues.apache.org/jira/browse/HDFS-13489 Project: Hadoop HDFS Issue Type: Task Components: hdfs Reporter: Harkrishn Patro Currently, hdfs only lists the snapshotable paths in the filesystem. This feature would add the functionality of figuring out if a given path is snapshotable or not. If yes, it would return the base snapshotable path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9518) LeaseRenewer - Improve Renew Method
[ https://issues.apache.org/jira/browse/HDFS-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445917#comment-16445917 ] BELUGA BEHR commented on HDFS-9518: --- Findbugs warning does not apply here. The Apache Commons Collection MultiMap being used here has different behaviors than a typical Map class. Please consider this patch for inclusion into the project. > LeaseRenewer - Improve Renew Method > --- > > Key: HDFS-9518 > URL: https://issues.apache.org/jira/browse/HDFS-9518 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HDFS-9518.2.patch, HDFS-9518.3.patch, LeaseRenewer.patch > > > Replaced the current implementation of > org.apache.hadoop.hdfs.client.impl.LeaseRenewer#renew() -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9518) LeaseRenewer - Improve Renew Method
[
https://issues.apache.org/jira/browse/HDFS-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445910#comment-16445910
]
genericqa commented on HDFS-9518:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs-client: The patch
generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 28s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
34s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
20s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | org.apache.hadoop.hdfs.client.impl.LeaseRenewer.renew() makes inefficient
use of keySet iterator instead of entrySet iterator At
LeaseRenewer.java:keySet iterator instead of entrySet iterator At
LeaseRenewer.java:[line 396] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-9518 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920027/HDFS-9518.3.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2c094b9cce1b 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 860cc28 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/24017/artifact/out/new-findbugs-hadoop-hdfs-projec
[jira] [Commented] (HDFS-9593) BlockTokenIdentifier Code Review
[
https://issues.apache.org/jira/browse/HDFS-9593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445871#comment-16445871
]
genericqa commented on HDFS-9593:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-client: The
patch generated 1 new + 5 unchanged - 1 fixed = 6 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
48s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-9593 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920011/HDFS-9593.3.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2fd9e6176a63 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / da5bcf5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/24016/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24016/testReport/ |
| Max. process+thread count | 302 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-
[jira] [Created] (HDFS-13488) RBF: Reject requests when a Router is overloaded
Íñigo Goiri created HDFS-13488: -- Summary: RBF: Reject requests when a Router is overloaded Key: HDFS-13488 URL: https://issues.apache.org/jira/browse/HDFS-13488 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Íñigo Goiri Assignee: Íñigo Goiri A Router might be overloaded when handling special cases (e.g. a slow subcluster). The Router could reject the requests and the client could try with another Router. We should leverage the Standby mechanism for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9518) LeaseRenewer - Improve Renew Method
[ https://issues.apache.org/jira/browse/HDFS-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445838#comment-16445838 ] BELUGA BEHR commented on HDFS-9518: --- Attached a new patch which fixes the unit test. Also removed some checkstyle warnings from the test class. Added in a JavaDoc to explain the renewal process (I know I found the previous code confusing) > LeaseRenewer - Improve Renew Method > --- > > Key: HDFS-9518 > URL: https://issues.apache.org/jira/browse/HDFS-9518 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HDFS-9518.2.patch, HDFS-9518.3.patch, LeaseRenewer.patch > > > Replaced the current implementation of > org.apache.hadoop.hdfs.client.impl.LeaseRenewer#renew() -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10943) rollEditLog expects empty EditsDoubleBuffer.bufCurrent which is not guaranteed
[
https://issues.apache.org/jira/browse/HDFS-10943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445837#comment-16445837
]
He Xiaoqiao commented on HDFS-10943:
hi [~daryn], I wonder is there any condition to make sure that
{{EditsDoubleBuffer.bufReady}} is empty when swap buffer if disable assertions.
Thanks for your help.
> rollEditLog expects empty EditsDoubleBuffer.bufCurrent which is not guaranteed
> --
>
> Key: HDFS-10943
> URL: https://issues.apache.org/jira/browse/HDFS-10943
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Priority: Major
>
> Per the following trace stack:
> {code}
> FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: finalize log
> segment 10562075963, 10562174157 failed for required journal
> (JournalAndStream(mgr=QJM to [0.0.0.1:8485, 0.0.0.2:8485, 0.0.0.3:8485,
> 0.0.0.4:8485, 0.0.0.5:8485], stream=QuorumOutputStream starting at txid
> 10562075963))
> java.io.IOException: FSEditStream has 49708 bytes still to be flushed and
> cannot be closed.
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer.close(EditsDoubleBuffer.java:66)
> at
> org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.close(QuorumOutputStream.java:65)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalAndStream.closeStream(JournalSet.java:115)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$4.apply(JournalSet.java:235)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:231)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:1243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:6437)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:1002)
> at
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:142)
> at
> org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:12025)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
> 2016-09-23 21:40:59,618 WARN
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Aborting
> QuorumOutputStream starting at txid 10562075963
> {code}
> The exception is from EditsDoubleBuffer
> {code}
> public void close() throws IOException {
> Preconditions.checkNotNull(bufCurrent);
> Preconditions.checkNotNull(bufReady);
> int bufSize = bufCurrent.size();
> if (bufSize != 0) {
> throw new IOException("FSEditStream has " + bufSize
> + " bytes still to be flushed and cannot be closed.");
> }
> IOUtils.cleanup(null, bufCurrent, bufReady);
> bufCurrent = bufReady = null;
> }
> {code}
> We can see that FSNamesystem.rollEditLog expects
> EditsDoubleBuffer.bufCurrent to be empty.
> Edits are recorded via FSEditLog$logSync, which does:
> {code}
>* The data is double-buffered within each edit log implementation so that
>* in-memory writing can occur in parallel with the on-disk writing.
>*
>* Each sync occurs in three steps:
>* 1. synchronized, it swaps the double buffer and sets the isSyncRunning
>* flag.
>* 2. unsynchronized, it flushes the data to storage
>* 3. synchronized, it resets the flag and notifies anyone waiting on the
>* sync.
>*
>* The lack of synchronization on step 2 allows other threads to continue
>* to write into the memory buffer while the sync is in progress.
>* Because this step is unsynchronized, actions that need to avoid
>* concurrency with sync() should be synchronized and also call
>* waitForSyncToFinish() before assuming they are running al
[jira] [Updated] (HDFS-9518) LeaseRenewer - Improve Renew Method
[ https://issues.apache.org/jira/browse/HDFS-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HDFS-9518: -- Attachment: HDFS-9518.3.patch > LeaseRenewer - Improve Renew Method > --- > > Key: HDFS-9518 > URL: https://issues.apache.org/jira/browse/HDFS-9518 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HDFS-9518.2.patch, HDFS-9518.3.patch, LeaseRenewer.patch > > > Replaced the current implementation of > org.apache.hadoop.hdfs.client.impl.LeaseRenewer#renew() -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13484) RBF: Disable Nameservices from the federation
[ https://issues.apache.org/jira/browse/HDFS-13484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445827#comment-16445827 ] Íñigo Goiri commented on HDFS-13484: Thanks [~linyiqun], let me add a separate JIRA for the overload case, it should be pretty isolated. > RBF: Disable Nameservices from the federation > - > > Key: HDFS-13484 > URL: https://issues.apache.org/jira/browse/HDFS-13484 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13484.000.patch > > > HDFS-13478 introduced the Decommission store. We should disable the access to > decommissioned subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13478) RBF: Disabled Nameservice store API
[ https://issues.apache.org/jira/browse/HDFS-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13478: --- Summary: RBF: Disabled Nameservice store API (was: RBF: Decommission store API) > RBF: Disabled Nameservice store API > --- > > Key: HDFS-13478 > URL: https://issues.apache.org/jira/browse/HDFS-13478 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13478.000.patch, HDFS-13478.001.patch, > HDFS-13478.002.patch, HDFS-13478.003.patch, HDFS-13478.004.patch > > > We have a subcluster in our federation that is for testing and is > missbehaving. This has a negative impact on the performance with operations > that go to every subcluster (e.g., renewLease() or setSafeMode()). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13484) RBF: Disable Nameservices from the federation
[ https://issues.apache.org/jira/browse/HDFS-13484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13484: --- Summary: RBF: Disable Nameservices from the federation (was: RBF: Decommission subclusters from the federation) > RBF: Disable Nameservices from the federation > - > > Key: HDFS-13484 > URL: https://issues.apache.org/jira/browse/HDFS-13484 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13484.000.patch > > > HDFS-13478 introduced the Decommission store. We should disable the access to > decommissioned subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9593) BlockTokenIdentifier Code Review
[ https://issues.apache.org/jira/browse/HDFS-9593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445725#comment-16445725 ] BELUGA BEHR commented on HDFS-9593: --- New patch to fix the checkstyle warning... > BlockTokenIdentifier Code Review > > > Key: HDFS-9593 > URL: https://issues.apache.org/jira/browse/HDFS-9593 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: BlockTokenIdentifier.HDFS-9593.patch, HDFS-9593.2.patch, > HDFS-9593.3.patch > > > Please find attached a code review of BlockTokenIdentifier. > The only real change is using an explicit StringBuilder instead of an > implicit one so that the internal buffer may be set more appropriately and > checking for zero-length string instead of comparing to an empty string. The > relatively small default buffer of 16 bytes would need to be expanded several > times to fit the incoming string. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9593) BlockTokenIdentifier Code Review
[ https://issues.apache.org/jira/browse/HDFS-9593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HDFS-9593: -- Attachment: HDFS-9593.3.patch > BlockTokenIdentifier Code Review > > > Key: HDFS-9593 > URL: https://issues.apache.org/jira/browse/HDFS-9593 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: BlockTokenIdentifier.HDFS-9593.patch, HDFS-9593.2.patch, > HDFS-9593.3.patch > > > Please find attached a code review of BlockTokenIdentifier. > The only real change is using an explicit StringBuilder instead of an > implicit one so that the internal buffer may be set more appropriately and > checking for zero-length string instead of comparing to an empty string. The > relatively small default buffer of 16 bytes would need to be expanded several > times to fit the incoming string. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13245) RBF: State store DBMS implementation
[ https://issues.apache.org/jira/browse/HDFS-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445655#comment-16445655 ] maobaolong commented on HDFS-13245: --- This patch seems good overall, i will take a look when i have time. Thank you for this contribution first. > RBF: State store DBMS implementation > > > Key: HDFS-13245 > URL: https://issues.apache.org/jira/browse/HDFS-13245 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: maobaolong >Assignee: Yiran Wu >Priority: Major > Attachments: HDFS-13245.001.patch, HDFS-13245.002.patch > > > Add a DBMS implementation for the State Store. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13408) MiniDFSCluster to support being built on randomized base directory
[
https://issues.apache.org/jira/browse/HDFS-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445615#comment-16445615
]
Xiao Liang commented on HDFS-13408:
---
Thank you [~giovanni.fumarola] for the suggestion, and thank you
[~chris.douglas] for adding more problem description, it's correct and precise,
the updated patch looks good to me.
{quote}Would the failing tests also fail to run in parallel, even on
non-Windows platforms?
{quote}
I don't have an environment to test running in parallel on non-Windows
platforms, but there should be file read/write conflicts between test cases
using the same base dir and test data file names, if running in parallel.
[~elgoiri] for the test cases, the 3 failed ones in the latest run don't seem
to relate to this change, as the default path will be the same as before, might
be random failure or caused by other changes?
> MiniDFSCluster to support being built on randomized base directory
> --
>
> Key: HDFS-13408
> URL: https://issues.apache.org/jira/browse/HDFS-13408
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Xiao Liang
>Assignee: Xiao Liang
>Priority: Major
> Labels: windows
> Attachments: HDFS-13408.000.patch, HDFS-13408.001.patch,
> HDFS-13408.002.patch
>
>
> Generated files of MiniDFSCluster during test are not properly cleaned in
> Windows, which fails all subsequent test cases using the same default
> directory (Windows does not allow other processes to delete them). By
> migrating to randomized base directories, the conflict of test path of test
> cases will be avoided, even if they are running at the same time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13165) [SPS]: Collects successfully moved block details via IBR
[
https://issues.apache.org/jira/browse/HDFS-13165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445593#comment-16445593
]
Rakesh R commented on HDFS-13165:
-
{quote}I've asked a few times why a new command (ie. newest name in this patch
is DNA_MOVEBLOCK) is required to move blocks across storages instead of using
the existing \{{DNA_TRANSFER? }}I've heard mention of delHint issues, but I'm
unclear why it's an issue?
{quote}
Thank you again [~daryn] for explaining {{delHint}} design thoughts. I
understand your point and completely agree with you, {{delHint}} is not
explicitly required in move block and is not a differentiate factor.
{quote}Sure, DNA_TRANSFER could be optimized to short-circuit for local moves,
{quote}
Yes, this functionality can be easily supported.
{quote}What am I missing?
{quote}
DNA_TRANSFER doesn't have ERROR_BLOCK_PINNED checks. Does it make sense to add
this check? This is again an optimization to avoid/reduce block move retries,
which is available with Mover tool.
I agree to use the {{DNA_TRANSFER}} for the *internal* sps. I have tried
prototyping DNA_TRANSFER functionality to *external* sps (sits outside
Namenode), here it requires a complex
[Sender#writeBlock|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java#L121]
function call with many arguments, do you agree to use the existing *Mover*
protocol
[Sender#replaceBlock|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java#L241]
for the *external* sps. With this proposal, I understand both
external/internal have different function calls, but my attempt is to simplify
the code changes.
Secondly, how about refactoring sps code to support DNA_TRANSFER protocol
changes to a separate jira sub-task, then continue implementing there. I will
create a patch with only the IBR related changes in this jira and then commit
it, that would help me rather than carrying bigger patch here.
> [SPS]: Collects successfully moved block details via IBR
>
>
> Key: HDFS-13165
> URL: https://issues.apache.org/jira/browse/HDFS-13165
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Attachments: HDFS-13165-HDFS-10285-00.patch,
> HDFS-13165-HDFS-10285-01.patch, HDFS-13165-HDFS-10285-02.patch,
> HDFS-13165-HDFS-10285-03.patch, HDFS-13165-HDFS-10285-04.patch,
> HDFS-13165-HDFS-10285-05.patch, HDFS-13165-HDFS-10285-06.patch,
> HDFS-13165-HDFS-10285-07.patch, HDFS-13165-HDFS-10285-08.patch,
> HDFS-13166-HDFS-10285-07.patch
>
>
> This task to make use of the existing IBR to get moved block details and
> remove unwanted future tracking logic exists in BlockStorageMovementTracker
> code, this is no more needed as the file level tracking maintained at NN
> itself.
> Following comments taken from HDFS-10285,
> [here|https://issues.apache.org/jira/browse/HDFS-10285?focusedCommentId=16347472&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16347472]
> Comment-3)
> {quote}BPServiceActor
> Is it actually sending back the moved blocks? Aren’t IBRs sufficient?{quote}
> Comment-21)
> {quote}
> BlockStorageMovementTracker
> Many data structures are riddled with non-threadsafe race conditions and risk
> of CMEs.
> Ex. The moverTaskFutures map. Adding new blocks and/or adding to a block's
> list of futures is synchronized. However the run loop does an unsynchronized
> block get, unsynchronized future remove, unsynchronized isEmpty, possibly
> another unsynchronized get, only then does it do a synchronized remove of the
> block. The whole chunk of code should be synchronized.
> Is the problematic moverTaskFutures even needed? It's aggregating futures
> per-block for seemingly no reason. Why track all the futures at all instead
> of just relying on the completion service? As best I can tell:
> It's only used to determine if a future from the completion service should be
> ignored during shutdown. Shutdown sets the running boolean to false and
> clears the entire datastructure so why not use the running boolean like a
> check just a little further down?
> As synchronization to sleep up to 2 seconds before performing a blocking
> moverCompletionService.take, but only when it thinks there are no active
> futures. I'll ignore the missed notify race that the bounded wait masks, but
> the real question is why not just do the blocking take?
> Why all the complexity? Am I missing something?
> BlocksMovementsStatusHandler
> Suffers same type of thread safety issues as StoragePolicySatisfyWorker. Ex.
> blockIdVsMovementStatus is
[jira] [Updated] (HDFS-13474) Unable to start Hadoop Name- and DataNodes
[
https://issues.apache.org/jira/browse/HDFS-13474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
robbie updated HDFS-13474:
--
Summary: Unable to start Hadoop Name- and DataNodes (was: Unable to start
Hadoop DataNodes)
> Unable to start Hadoop Name- and DataNodes
> --
>
> Key: HDFS-13474
> URL: https://issues.apache.org/jira/browse/HDFS-13474
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: robbie
>Priority: Major
> Attachments: hadoop-roycecoll...@steelydan.com-datanode-c0315.log,
> hadoop-roycecoll...@steelydan.com-datanode-c0315.out,
> hadoop-roycecoll...@steelydan.com-namenode-c0315.log,
> hadoop-roycecoll...@steelydan.com-namenode-c0315.out,
> hadoop-roycecoll...@steelydan.com-secondarynamenode-c0315.log,
> hadoop-roycecoll...@steelydan.com-secondarynamenode-c0315.out
>
>
> I am trying to follow the instructions in the Getting Started guide,
> [http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html#YARN_on_Single_Node]
> I have confirmed, that I can `ssh localhost` without a password prompt. I
> have also run the following steps,
> {quote}1. $ bin/hdfs namenode -format
> 2. $ sbin/start-dfs.sh
> {quote}
> But I cant run step 3. to browse the location at `[http://localhost:9870/]`.
> When I run `>jsp` from the terminal prompt I just get returned,
> {quote}14900 Jps
> {quote}
> I was expecting a list of my nodes.
> In the Logs I see two error messages towards the end,
> {quote}2018-04-18 14:15:42,516 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: RECEIVED SIGNAL 15: SIGTERM
> {quote}
> {quote}2018-04-18 14:15:42,516 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: RECEIVED SIGNAL 1: SIGHUP
> {quote}
> {quote}2018-04-18 14:15:42,517 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down DataNode at c0315/127.0.1.1
> /
> {quote}
> I will attach the full logs with this bug report.
> Can anyone help even with ways to debug this please ?
>
> Java Version,
> {quote}rcoll...@steelydan.com@c0315:~/temp/logs/hadoop$ java --version
> java 9.0.4
> Java(TM) SE Runtime Environment (build 9.0.4+11)
> Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode)
> {quote}
> EDIT1 : following Brahma Reddy's suggestion I have repeated the steps with
> Java8 as well and get the same error message.
> Ubuntu version,
> {quote}$ lsb_release -a
> No LSB modules are available.
> Distributor ID: neon
> Description: KDE neon User Edition 5.12
> Release: 16.04
> Codename: xenial
> {quote}
> I have tried running the commands, `bin/hdfs version`
> {quote}Hadoop 3.1.0
> Source code repository [https://github.com/apache/hadoop] -r
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> Compiled by centos on 2018-03-30T00:00Z
> Compiled with protoc 2.5.0
> From source with checksum 14182d20c972b3e2105580a1ad6990
> This command was run using
> /home/steelydan.com/roycecollige/Apps/hadoop-3.1.0/share/hadoop/common/hadoop-common-3.1.0.jar
> {quote}
> when I try `bin/hdfs groups` it doesnt return but gives me,
> {quote}018-04-18 15:33:34,590 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:9000. Already tried 0 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
> {quote}
> when I try, `$ bin/hdfs lsSnapshottableDir`
> {quote}lsSnapshottableDir: Call From c0315/127.0.1.1 to localhost:9000 failed
> on connection exception: java.net.ConnectException: Connection refused; For
> more details see: [http://wiki|http://wiki/].
> apache.org/hadoop/ConnectionRefused
> {quote}
>
> when I try, `$ bin/hdfs classpath`
> {quote}/home/steelydan.com/roycecollige/Apps/hadoop-3.1.0/etc/hadoop:/home/steelydan.com/roycecollige/Apps/hadoop-3.1.0/share/hadoop/common/lib/*:/home/steelydan.com/roycecollige/Apps/hadoop-3.1.0/share/hadoop/common/*:/home/steelydan.com/roycecollige/Apps/hadoop-3.1.0/share/hadoop/hdfs:/home/steelydan.com/roycecollige/Apps/hadoop-3.1.0/share/hadoop/hdfs/lib/*:/home/steelydan.com/roycecollige/Apps/hadoop-3.1.0/share/hadoop/hdfs/*:/home/steelydan.com/roycecollige/Apps/hadoop-3.1.0/share/hadoop/mapreduce/*:/home/steelydan.com/roycecollige/Apps/hadoop-3.1.0/share/hadoop/yarn:/home/steelydan.com/roycecollige/Apps/hadoop-3.1.0/share/hadoop/yarn/lib/*:/home/steelydan.com/roycecollige/Apps/hadoop-3.1.0/share/hadoop/yarn/*
> {quote}
> core-site.xml
> {quote}
>
>
> fs.defaultFS
> hdfs://localhost:9000
>
>
> {quote}
>
> hdfs-site.xml
> {quote}
>
> dfs.replication
> 1
>
>
> {quote}
> mapred-site.xml
> {quote}
>
> mapreduce.framework.name
> yarn
>
>
> {quote}
--
Thi
[jira] [Commented] (HDFS-13484) RBF: Decommission subclusters from the federation
[
https://issues.apache.org/jira/browse/HDFS-13484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445467#comment-16445467
]
Yiqun Lin commented on HDFS-13484:
--
Thanks for attaching the patch, [~elgoiri]. Haven't taken a deep review, some
initial comments:
* This line looks unnecessary {{System.out.println("IN: " +
location.getDestinations());}}
* In the patch, I saw lots of change for router overload control. Can we move
this logic into other JIRA if we want to make this change? This will make this
patch clear.
> RBF: Decommission subclusters from the federation
> -
>
> Key: HDFS-13484
> URL: https://issues.apache.org/jira/browse/HDFS-13484
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-13484.000.patch
>
>
> HDFS-13478 introduced the Decommission store. We should disable the access to
> decommissioned subclusters.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13486) Backport HDFS-11817 to branch-2.7
[
https://issues.apache.org/jira/browse/HDFS-13486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445460#comment-16445460
]
genericqa commented on HDFS-13486:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.7 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} branch-2.7 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 465 unchanged - 1 fixed = 466 total (was 466) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 60 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}155m 1s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 2m
4s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-hdfs:41 |
| Timed out junit tests | org.apache.hadoop.hdfs.TestEncryptionZones |
| | org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA |
| | org.apache.hadoop.hdfs.qjournal.client.TestQuorumJournalManager |
| | org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | org.apache.hadoop.hdfs.TestDatanodeRegistration |
| | org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | org.apache.hadoop.hdfs.qjournal.server.TestJournalNode |
| | org.apache.hadoop.hdfs.TestQuota |
| | org.apache.hadoop.hdfs.TestDataTransferKeepalive |
| | org.apache.hadoop.hdfs.TestDatanodeDeath |
| | org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart |
| | org.apache.hadoop.hdfs.TestFileAppend |
| | org.apache.hadoop.hdfs.TestPread |
| | org.apache.hadoop.hdfs.TestFileAppend4 |
| | org.apache.hadoop.hdfs.TestDFSFinalize |
| | org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults |
| | org.apache.hadoop.hdfs.TestDFSUpgradeFromImage |
| | org.apache.hadoop.hdfs.web.TestWebHdfsTokens |
| | org.apache.hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs |
| | org.apache.hadoop.hdfs.TestFileCreationDelete |
| | org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS |
| | org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer |
| | org.apache.hadoop.hdfs.TestBlockStoragePolicy |
| | org.apache.hadoop.hdfs.TestRollingUpgrade |
| | org.apache.hadoop.hdfs.TestLease |
| | org.apache.hadoop.hdfs.qjournal.TestNNWithQJM |
| | org.apache.hadoop.hdfs.web.TestWebHDFS |
| | org.apache.ha
[jira] [Commented] (HDFS-13283) Percentage based Reserved Space Calculation for DataNode
[
https://issues.apache.org/jira/browse/HDFS-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445441#comment-16445441
]
genericqa commented on HDFS-13283:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 14s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13283 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919954/HDFS-13283.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux fc44fe564b7b 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / da5bcf5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24014/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Tes
[jira] [Commented] (HDFS-13408) MiniDFSCluster to support being built on randomized base directory
[
https://issues.apache.org/jira/browse/HDFS-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445400#comment-16445400
]
genericqa commented on HDFS-13408:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}171m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13408 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919948/HDFS-13408.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 224b176cbd63 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / da5bcf5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24013/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24013/testReport/ |
| Max. process+thread count | 2862 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console
