[jira] [Created] (HDFS-10883) `getTrashRoot`'s behavior is not consistent in DFS after enabling EZ.
Yuanbo Liu created HDFS-10883:
-
Summary: `getTrashRoot`'s behavior is not consistent in DFS after
enabling EZ.
Key: HDFS-10883
URL: https://issues.apache.org/jira/browse/HDFS-10883
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Yuanbo Liu
Assignee: Yuanbo Liu
Let's say root path ("/") is the encryption zone, and there is a file called
"/test" in root path.
{code}
dfs.getTrashRoot(new Path("/"))
{code}
returns "/user/$USER/.Trash",
while
{code}
dfs.getTrashRoot(new Path("/test"))
{code}
returns "/.Trash/$USER".
The second behavior is not correct. Since root path is the encryption zone,
which means all files/directories in DFS are encrypted, it's more reasonable
to return "/user/$USER/.Trash" no matter what the path is.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10882) Federation State Store Interface API
[
https://issues.apache.org/jira/browse/HDFS-10882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508793#comment-15508793
]
Hadoop QA commented on HDFS-10882:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
19s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 20 new + 0 unchanged - 0 fixed = 20 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
51s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
52s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7
unchanged - 0 fixed = 8 total (was 7) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Exception is caught when Exception is not thrown in
org.apache.hadoop.hdfs.server.federation.store.protocol.impl.pb.FederationProtocolPBTranslator.getBuilder()
At FederationProtocolPBTranslator.java:is not thrown in
org.apache.hadoop.hdfs.server.federation.store.protocol.impl.pb.FederationProtocolPBTranslator.getBuilder()
At FederationProtocolPBTranslator.java:[line 71] |
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10882 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829493/HDFS-10882-HDFS-10467-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2afc9c6af2ea 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-10467 / 382dff7 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Resolved] (HDFS-9333) Some tests using MiniDFSCluster errored complaining port in use
[
https://issues.apache.org/jira/browse/HDFS-9333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HDFS-9333.
---
Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
2.8.0
Resolving since it looks like this was committed to trunk, branch-2,
branch-2.8. Thanks for working on this [~iwasakims]!
> Some tests using MiniDFSCluster errored complaining port in use
> ---
>
> Key: HDFS-9333
> URL: https://issues.apache.org/jira/browse/HDFS-9333
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Reporter: Kai Zheng
>Assignee: Masatake Iwasaki
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-9333.001.patch, HDFS-9333.002.patch,
> HDFS-9333.003.patch
>
>
> Ref. the following:
> {noformat}
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.483 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
> testRead(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped)
> Time elapsed: 11.021 sec <<< ERROR!
> java.net.BindException: Port in use: localhost:49333
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> at
> org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:884)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:826)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:821)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:675)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2015)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1996)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS.doTestRead(TestBlockTokenWithDFS.java:539)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped.testRead(TestBlockTokenWithDFSStriped.java:62)
> {noformat}
> Another one:
> {noformat}
> Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 9.859 sec <<<
> FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController
> testFailoverAndBackOnNNShutdown(org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController)
> Time elapsed: 0.41 sec <<< ERROR!
> java.net.BindException: Problem binding to [localhost:10021]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:399)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1245)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1014)
> at
>
[jira] [Commented] (HDFS-10881) Federation State Store Driver API
[
https://issues.apache.org/jira/browse/HDFS-10881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508686#comment-15508686
]
Hadoop QA commented on HDFS-10881:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
6s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 12 new + 0 unchanged - 0 fixed = 12 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
47s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 60m
39s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Redundant nullcheck of results, which is known to be non-null in
org.apache.hadoop.hdfs.server.federation.store.driver.StateStoreBase.getSingle(Class,
Map) Redundant null check at StateStoreBase.java:is known to be non-null in
org.apache.hadoop.hdfs.server.federation.store.driver.StateStoreBase.getSingle(Class,
Map) Redundant null check at StateStoreBase.java:[line 115] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10881 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829488/HDFS-10881-HDFS-10467-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4a398a82bb9e 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-10467 / 382dff7 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16821/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
[jira] [Commented] (HDFS-10630) Federation State Store
[
https://issues.apache.org/jira/browse/HDFS-10630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508623#comment-15508623
]
Hadoop QA commented on HDFS-10630:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
17s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 66 new + 390 unchanged - 0 fixed = 456 total (was 390) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
53s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 20s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Useless control flow in
org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$MockEntityProto$Builder.maybeForceBuilderInitialization()
At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line
520] |
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.tools.TestHdfsConfigFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10630 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829486/HDFS-10630-HDFS-10467-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle cc |
| uname | Linux 9092ae0749f5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-10467 / 382dff7 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Updated] (HDFS-10882) Federation State Store Interface API
[ https://issues.apache.org/jira/browse/HDFS-10882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Kace updated HDFS-10882: -- Assignee: Jason Kace Status: Patch Available (was: Open) > Federation State Store Interface API > > > Key: HDFS-10882 > URL: https://issues.apache.org/jira/browse/HDFS-10882 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Jason Kace >Assignee: Jason Kace > Attachments: HDFS-10882-HDFS-10467-001.patch > > > The minimal classes and interfaces required to create state store internal > data APIs using protobuf serialization. This is a pre-requisite for higher > level APIs such as the registration API and the mount table API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10882) Federation State Store Interface API
[ https://issues.apache.org/jira/browse/HDFS-10882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Kace updated HDFS-10882: -- Attachment: HDFS-10882-HDFS-10467-001.patch Attaching the first patch version. This patch contains: 1) The abstract class that is the base for all data records (BaseRecord) 2) The abstract class that is the base for all API parameter (request/response) objects (FedeationStateStoreInterface) 3) Protobuf helpers to instantiate and populate data records and API objects > Federation State Store Interface API > > > Key: HDFS-10882 > URL: https://issues.apache.org/jira/browse/HDFS-10882 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Jason Kace > Attachments: HDFS-10882-HDFS-10467-001.patch > > > The minimal classes and interfaces required to create state store internal > data APIs using protobuf serialization. This is a pre-requisite for higher > level APIs such as the registration API and the mount table API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10631) Federation State Store ZooKeeper implementation
[
https://issues.apache.org/jira/browse/HDFS-10631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508531#comment-15508531
]
Hadoop QA commented on HDFS-10631:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
35s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 29 new + 0 unchanged - 0 fixed = 29 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
2s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 7 new + 0
unchanged - 0 fixed = 7 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Found reliance on default encoding in
org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreZooKeeperImpl.writeNode(String,
String, boolean, boolean):in
org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreZooKeeperImpl.writeNode(String,
String, boolean, boolean): String.getBytes() At
StateStoreZooKeeperImpl.java:[line 402] |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreZooKeeperImpl.zkAuths;
locked 50% of time Unsynchronized access at StateStoreZooKeeperImpl.java:50%
of time Unsynchronized access at StateStoreZooKeeperImpl.java:[line 98] |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreZooKeeperImpl.zkHost;
locked 50% of time Unsynchronized access at StateStoreZooKeeperImpl.java:50%
of time Unsynchronized access at StateStoreZooKeeperImpl.java:[line 94] |
| | Redundant nullcheck of znode, which is known to be non-null in
org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreZooKeeperImpl.deleteAll(Class)
Redundant null check at StateStoreZooKeeperImpl.java:is known to be non-null
in
[jira] [Updated] (HDFS-10881) Federation State Store Driver API
[ https://issues.apache.org/jira/browse/HDFS-10881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Kace updated HDFS-10881: -- Assignee: Jason Kace Status: Patch Available (was: Open) > Federation State Store Driver API > - > > Key: HDFS-10881 > URL: https://issues.apache.org/jira/browse/HDFS-10881 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Jason Kace >Assignee: Jason Kace > Attachments: HDFS-10881-HDFS-10467-001.patch > > > The API interfaces and minimal classes required to support a state store data > backend such as ZooKeeper or a file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10881) Federation State Store Driver API
[ https://issues.apache.org/jira/browse/HDFS-10881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Kace updated HDFS-10881: -- Attachment: HDFS-10881-HDFS-10467-001.patch First version of the state store driver API patch. This patch contains the min number of interfaces and code to support creation of a state store driver backend. It also illustrates the general handling of data records by the driver. All future patches will be based on this code. > Federation State Store Driver API > - > > Key: HDFS-10881 > URL: https://issues.apache.org/jira/browse/HDFS-10881 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Jason Kace > Attachments: HDFS-10881-HDFS-10467-001.patch > > > The API interfaces and minimal classes required to support a state store data > backend such as ZooKeeper or a file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9333) Some tests using MiniDFSCluster errored complaining port in use
[
https://issues.apache.org/jira/browse/HDFS-9333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated HDFS-9333:
---
Status: In Progress (was: Patch Available)
> Some tests using MiniDFSCluster errored complaining port in use
> ---
>
> Key: HDFS-9333
> URL: https://issues.apache.org/jira/browse/HDFS-9333
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Reporter: Kai Zheng
>Assignee: Masatake Iwasaki
>Priority: Minor
> Attachments: HDFS-9333.001.patch, HDFS-9333.002.patch,
> HDFS-9333.003.patch
>
>
> Ref. the following:
> {noformat}
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.483 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
> testRead(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped)
> Time elapsed: 11.021 sec <<< ERROR!
> java.net.BindException: Port in use: localhost:49333
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> at
> org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:884)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:826)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:821)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:675)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2015)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1996)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS.doTestRead(TestBlockTokenWithDFS.java:539)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped.testRead(TestBlockTokenWithDFSStriped.java:62)
> {noformat}
> Another one:
> {noformat}
> Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 9.859 sec <<<
> FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController
> testFailoverAndBackOnNNShutdown(org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController)
> Time elapsed: 0.41 sec <<< ERROR!
> java.net.BindException: Problem binding to [localhost:10021]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:399)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1245)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1014)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:889)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:821)
> at
[jira] [Updated] (HDFS-10630) Federation State Store
[ https://issues.apache.org/jira/browse/HDFS-10630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Kace updated HDFS-10630: -- Attachment: HDFS-10630-HDFS-10467-003.patch Updating patch: 1) Findbugs errors that are relevant (not from protobuf or stubs) 2) License and whitespace issues 3) Updating patch naming convention > Federation State Store > -- > > Key: HDFS-10630 > URL: https://issues.apache.org/jira/browse/HDFS-10630 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Inigo Goiri >Assignee: Jason Kace > Attachments: HDFS-10630-HDFS-10467-003.patch, HDFS-10630.001.patch, > HDFS-10630.002.patch > > > Interface to store the federation shared state across Routers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9333) Some tests using MiniDFSCluster errored complaining port in use
[
https://issues.apache.org/jira/browse/HDFS-9333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508420#comment-15508420
]
Hudson commented on HDFS-9333:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10471 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10471/])
HDFS-9333. Some tests using MiniDFSCluster errored complaining port in
(iwasakims: rev 964e546ab1dba5f5d53b209ec6c9a70a85654765)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFS.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFSStriped.java
> Some tests using MiniDFSCluster errored complaining port in use
> ---
>
> Key: HDFS-9333
> URL: https://issues.apache.org/jira/browse/HDFS-9333
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Reporter: Kai Zheng
>Assignee: Masatake Iwasaki
>Priority: Minor
> Attachments: HDFS-9333.001.patch, HDFS-9333.002.patch,
> HDFS-9333.003.patch
>
>
> Ref. the following:
> {noformat}
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.483 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
> testRead(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped)
> Time elapsed: 11.021 sec <<< ERROR!
> java.net.BindException: Port in use: localhost:49333
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> at
> org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:884)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:826)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:821)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:675)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2015)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1996)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS.doTestRead(TestBlockTokenWithDFS.java:539)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped.testRead(TestBlockTokenWithDFSStriped.java:62)
> {noformat}
> Another one:
> {noformat}
> Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 9.859 sec <<<
> FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController
> testFailoverAndBackOnNNShutdown(org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController)
> Time elapsed: 0.41 sec <<< ERROR!
> java.net.BindException: Problem binding to [localhost:10021]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:399)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at
>
[jira] [Created] (HDFS-10882) Federation State Store Interface API
Jason Kace created HDFS-10882: - Summary: Federation State Store Interface API Key: HDFS-10882 URL: https://issues.apache.org/jira/browse/HDFS-10882 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jason Kace The minimal classes and interfaces required to create state store internal data APIs using protobuf serialization. This is a pre-requisite for higher level APIs such as the registration API and the mount table API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10881) Federation State Store Driver API
Jason Kace created HDFS-10881: - Summary: Federation State Store Driver API Key: HDFS-10881 URL: https://issues.apache.org/jira/browse/HDFS-10881 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jason Kace The API interfaces and minimal classes required to support a state store data backend such as ZooKeeper or a file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10861) Refactor StripeReaders and use ECChunk version decode API
[ https://issues.apache.org/jira/browse/HDFS-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508390#comment-15508390 ] SammiChen commented on HDFS-10861: -- Thanks Kai for take time review the patch and commit it! > Refactor StripeReaders and use ECChunk version decode API > - > > Key: HDFS-10861 > URL: https://issues.apache.org/jira/browse/HDFS-10861 > Project: Hadoop HDFS > Issue Type: Task >Reporter: SammiChen >Assignee: SammiChen > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10861-v1.patch, HDFS-10861-v2.patch, > HDFS-10861-v3.patch, HDFS-10861-v4.patch, HDFS-10861-v5.patch, > HDFS-10861-v6.patch > > > Refactor StatefulStripeReader and PositionStripeReader, use ECChunk version > decode API. After the refactor, it is approaching very near now to the ideal > state desired by next step, employing ErasureCoder API instead of > RawErasureCoder API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
[
https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508343#comment-15508343
]
Hadoop QA commented on HDFS-10757:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
20s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10757 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829469/HDFS-10757.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b622f117e0be 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5a58bfe |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16818/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16818/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> KMSClientProvider combined with KeyProviderCache can result in wrong UGI
> being used
> ---
>
> Key: HDFS-10757
> URL: https://issues.apache.org/jira/browse/HDFS-10757
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Xiaoyu Yao
>Priority: Critical
> Attachments: HDFS-10757.00.patch,
[jira] [Commented] (HDFS-10687) Federation Membership State Store internal APIs
[
https://issues.apache.org/jira/browse/HDFS-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508341#comment-15508341
]
Hadoop QA commented on HDFS-10687:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-build-tools {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 50s{color} | {color:orange} root: The patch generated 78 new + 0 unchanged -
0 fixed = 78 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-build-tools {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
58s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 11 new + 0
unchanged - 0 fixed = 11 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
2s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 11 new + 7
unchanged - 0 fixed = 18 total (was 7) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
10s{color} | {color:green} hadoop-build-tools in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
23s{color} | {color:red} The patch generated 4 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Useless control flow in
org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$FederationNamespaceInfoProto$Builder.maybeForceBuilderInitialization()
At
[jira] [Updated] (HDFS-10631) Federation State Store ZooKeeper implementation
[ https://issues.apache.org/jira/browse/HDFS-10631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Kace updated HDFS-10631: -- Assignee: Jason Kace Status: Patch Available (was: Open) > Federation State Store ZooKeeper implementation > --- > > Key: HDFS-10631 > URL: https://issues.apache.org/jira/browse/HDFS-10631 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Inigo Goiri >Assignee: Jason Kace > Attachments: HDFS-10631-HDFS-10467-001.patch > > > State Store implementation using ZooKeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10631) Federation State Store ZooKeeper implementation
[ https://issues.apache.org/jira/browse/HDFS-10631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Kace updated HDFS-10631: -- Attachment: HDFS-10631-HDFS-10467-001.patch Attaching first patch for the ZK state store backend. I stubbed out most of the dependent code, the full code for these classes can be found in HDFS-10630. This patch is focused on the ZK driver which reads and writes records to a ZK data store. > Federation State Store ZooKeeper implementation > --- > > Key: HDFS-10631 > URL: https://issues.apache.org/jira/browse/HDFS-10631 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Inigo Goiri > Attachments: HDFS-10631-HDFS-10467-001.patch > > > State Store implementation using ZooKeeper. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10879) TestEncryptionZonesWithKMS#testReadWrite fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508312#comment-15508312
]
Hudson commented on HDFS-10879:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10470 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10470/])
HDFS-10879. TestEncryptionZonesWithKMS#testReadWrite fails (xiao: rev
0e918dff594e9ba5434fdee7fc1f6394b62b32cd)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
> TestEncryptionZonesWithKMS#testReadWrite fails intermittently
> -
>
> Key: HDFS-10879
> URL: https://issues.apache.org/jira/browse/HDFS-10879
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Labels: test
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10879.01.patch
>
>
> {noformat}
> Error Message:
> Key was rolled, versions should be different. Actual: test_key@0
> Stack Trace:
> java.lang.AssertionError: Key was rolled, versions should be different.
> Actual: test_key@0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failEquals(Assert.java:185)
> at org.junit.Assert.assertNotEquals(Assert.java:161)
> at
> org.apache.hadoop.hdfs.TestEncryptionZones.testReadWrite(TestEncryptionZones.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10879) TestEncryptionZonesWithKMS#testReadWrite fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-10879:
-
Summary: TestEncryptionZonesWithKMS#testReadWrite fails intermittently
(was: TestEncryptionZonesWithKMS#testReadWrite fails intermitently)
> TestEncryptionZonesWithKMS#testReadWrite fails intermittently
> -
>
> Key: HDFS-10879
> URL: https://issues.apache.org/jira/browse/HDFS-10879
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Labels: test
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10879.01.patch
>
>
> {noformat}
> Error Message:
> Key was rolled, versions should be different. Actual: test_key@0
> Stack Trace:
> java.lang.AssertionError: Key was rolled, versions should be different.
> Actual: test_key@0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failEquals(Assert.java:185)
> at org.junit.Assert.assertNotEquals(Assert.java:161)
> at
> org.apache.hadoop.hdfs.TestEncryptionZones.testReadWrite(TestEncryptionZones.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10630) Federation State Store
[
https://issues.apache.org/jira/browse/HDFS-10630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508189#comment-15508189
]
Hadoop QA commented on HDFS-10630:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 66 new + 394 unchanged - 0 fixed = 460 total (was 394) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
10s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 5 new + 0
unchanged - 0 fixed = 5 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
20s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Useless control flow in
org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$MockEntityProto$Builder.maybeForceBuilderInitialization()
At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line
520] |
| | Redundant nullcheck of results, which is known to be non-null in
org.apache.hadoop.hdfs.server.federation.store.driver.StateStoreBase.getSingle(Class,
Map) Redundant null check at StateStoreBase.java:is known to be non-null in
org.apache.hadoop.hdfs.server.federation.store.driver.StateStoreBase.getSingle(Class,
Map) Redundant null check at StateStoreBase.java:[line 161] |
| | Found reliance on default encoding in
org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreFileImpl.getReader(Class):in
org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreFileImpl.getReader(Class):
new java.io.FileReader(File) At StateStoreFileImpl.java:[line 108] |
| | Found reliance on default encoding in
org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreFileImpl.getWriter(Class):in
[jira] [Updated] (HDFS-10879) TestEncryptionZonesWithKMS#testReadWrite fails intermitently
[
https://issues.apache.org/jira/browse/HDFS-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-10879:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.8.0
Status: Resolved (was: Patch Available)
Committed this to trunk, branch-2 and branch-2.8. Thanks Andrew for the prompt
review!
> TestEncryptionZonesWithKMS#testReadWrite fails intermitently
>
>
> Key: HDFS-10879
> URL: https://issues.apache.org/jira/browse/HDFS-10879
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Labels: test
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10879.01.patch
>
>
> {noformat}
> Error Message:
> Key was rolled, versions should be different. Actual: test_key@0
> Stack Trace:
> java.lang.AssertionError: Key was rolled, versions should be different.
> Actual: test_key@0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failEquals(Assert.java:185)
> at org.junit.Assert.assertNotEquals(Assert.java:161)
> at
> org.apache.hadoop.hdfs.TestEncryptionZones.testReadWrite(TestEncryptionZones.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10314) A new tool to sync current HDFS view to specified snapshot
[
https://issues.apache.org/jira/browse/HDFS-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508167#comment-15508167
]
Yongjun Zhang commented on HDFS-10314:
--
Many thanks [~jingzhao] for the review and feedback. Please see my answers
below.
{quote}
So the current patch actually adds a new distsync extension and implements the
"calculating diff on target cluster" approach?
{quote}
Yes. This is the result of our discussion in HDFS-9820.
Though I preferred adding -rdiff as a symmetric behavior as how -diff works in
distcp, as reported in HDFS-9820, I think your suggestion of creating a new
tool is fine, as long as we leverage the code that does -diff in distcp, and
minimize code duplication.
{quote}
I think to have the diff calculated on target is fine,
{quote}
Yes. Since the goal is to make the target's state go to a specified snapshot,
we'd better calculate snapshot diff at the target.
{quote}
but I'm not sure to directly extend the current distcp is a good idea.
{quote}
There are couple of reasons when I came up with the idea of extending distcp:
* distsync is a customized distcp, it extends distcp's -diff behavior to
support -rdiff.
* it's better to re-use the code that implements -diff, extending allows
re-using the existing implementation of "-diff". You can see it's only 124
lines of code (including the header and imports) in DistSync.java in my patch
rev001.
{quote}
Correct me if I'm wrong. Here's my current understanding of the patch:
1. our main motivation is still to utilize distcp to restore a snapshot
2. the idea is to compute the delta on the target cluster, and for modified
files we get their original state from the source.
{quote}
Yes. However, for modified files, I intended to make it flexible to copy from
the specified snapshot of either the source or the target.
{quote}
In that sense, I think a simpler way is to wrap (but not extend) the current
distcp in the snapshot-restore tool:
1. The tool takes a single cluster and a target snapshot as arguments
2. The tool computes the delta for restoring using snapshot diff report
3. The tool does rename/delete etc. metadata ops to revert part of the diff
4. The tool uses the distcp (by invokes distcp as a library) to copy the
original states of modified files
In this way we can minimize the change (no need to touch the current distcp
implementation/arguments), and provides a new tool with simple but clear
semantic. We may lose some flexibility (only handling one cluster) but the tool
itself will be easy to use and will not cause any confusion to the end users.
What do you think? Please let me know if I miss anything.
{quote}
We discussed two overall solutions earlier.
* Solution A. What proposed in HDFS-9820: adding "-rdiff s2 s1" to distcp, to
achieve the symmetric behavior as "-diff s1 s2" of distcp.
* Solution B. What proposed in HDFS-10314: introducing a new tool, that allows
to sync a target cluster to a specified snapshot.
For Solution B, there are two approaches, one (B.1) is my patch rev001 here,
the other (B.2) is what you proposed above.
Some thoughts:
# Creating a new tool itself is going to mean extra support, that's why I
preferred solution #A, which is the simplest.
# Given that we want to create a new tool, we'd better maximize code sharing,
otherwise, it's going to be both more development effort and extra support
effort.
# To me, the way suggested by solution #B.2 disallows sharing the existing
implementation of -diff in distcp. Thus I think it's actually not simpler, and
would incur support burden for future because of the duplicated code.
# I think we agreed per our discussion that if we create a new tool, then you
don't have strong opinion whether we copy from a different cluster or from the
same target cluster. As I shared earlier, I can tell from the user's case, that
copying from a different mirror cluster can be much faster sometimes. So I kept
suggesting that it would be better to support the flexibility, to copy from
either the source or the target.
Would you please kindly share the specific problems you see with solution #B.1?
Honestly speaking, I still prefer solution #A. But I'm ok with solution B,
except I really hope to share the code of -diff implemented in distcp already.
Thanks a lot.
> A new tool to sync current HDFS view to specified snapshot
> --
>
> Key: HDFS-10314
> URL: https://issues.apache.org/jira/browse/HDFS-10314
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-10314.001.patch
>
>
> HDFS-9820 proposed adding -rdiff switch to distcp, as a reversed operation of
> -diff switch.
> Upon discussion with [~jingzhao], we will introduce a new tool
[jira] [Commented] (HDFS-10800) [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks which were placed in other storages than what NN is expecting.
[
https://issues.apache.org/jira/browse/HDFS-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508168#comment-15508168
]
Hadoop QA commented on HDFS-10800:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
28s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 7 new + 251 unchanged - 0 fixed = 258 total (was 251) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 59m
29s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10800 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829443/HDFS-10800-HDFS-10285-03.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1a59d518e964 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-10285 / 7e8c8e2 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16815/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16815/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16815/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks
> which were placed in other storages than what NN is expecting.
> ---
>
> Key: HDFS-10800
> URL:
[jira] [Commented] (HDFS-10824) MiniDFSCluster#storageCapacities has no effects on real capacity
[
https://issues.apache.org/jira/browse/HDFS-10824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508145#comment-15508145
]
Hadoop QA commented on HDFS-10824:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 205 unchanged - 1 fixed = 206 total (was 206) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 58m
6s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10824 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829442/HDFS-10824.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5ba6a4674d51 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e80386d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16814/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16814/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16814/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> MiniDFSCluster#storageCapacities has no effects on real capacity
>
>
> Key: HDFS-10824
> URL: https://issues.apache.org/jira/browse/HDFS-10824
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
[jira] [Updated] (HDFS-7530) Allow renaming of encryption zone roots
[
https://issues.apache.org/jira/browse/HDFS-7530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated HDFS-7530:
---
Target Version/s: 2.7.0, 2.6.5 (was: 2.7.0)
Adding 2.6.5 to the target versions. We are looking to backport this as part of
the 2.6.5 release. This is needed to fix the unit test added to 2.6.5 as part
of HDFS-9688.
[~sjlee0] I have tested the cherry-pick from branch-2.7 and it is clean, other
than changes.txt. I also ran {{TestNestedEncryptionZones}} and
{{TestEncryptionZones}} and both passed.
> Allow renaming of encryption zone roots
> ---
>
> Key: HDFS-7530
> URL: https://issues.apache.org/jira/browse/HDFS-7530
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
>Priority: Minor
> Fix For: 2.7.0
>
> Attachments: HDFS-7530.001.patch, HDFS-7530.002.patch,
> HDFS-7530.003.patch
>
>
> It should be possible to do
> hdfs dfs -mv /ezroot /newnameforezroot
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Attachment: (was: HADOOP-10757.02.patch) > KMSClientProvider combined with KeyProviderCache can result in wrong UGI > being used > --- > > Key: HDFS-10757 > URL: https://issues.apache.org/jira/browse/HDFS-10757 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Xiaoyu Yao >Priority: Critical > Attachments: HDFS-10757.00.patch, HDFS-10757.01.patch, > HDFS-10757.02.patch > > > ClientContext::get gets the context from CACHE via a config setting based > name, then KeyProviderCache stored in ClientContext gets the key provider > cached by URI from the configuration, too. These would return the same > KeyProvider regardless of current UGI. > KMSClientProvider caches the UGI (actualUgi) in ctor; that means in > particular that all the users of DFS with KMSClientProvider in a process will > get the KMS token (along with other credentials) of the first user, via the > above cache. > Either KMSClientProvider shouldn't store the UGI, or one of the caches should > be UGI-aware, like the FS object cache. > Side note: the comment in createConnection that purports to handle the > different UGI doesn't seem to cover what it says it covers. In our case, we > have two unrelated UGIs with no auth (createRemoteUser) with bunch of tokens, > including a KMS token, added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Attachment: HDFS-10757.02.patch > KMSClientProvider combined with KeyProviderCache can result in wrong UGI > being used > --- > > Key: HDFS-10757 > URL: https://issues.apache.org/jira/browse/HDFS-10757 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Xiaoyu Yao >Priority: Critical > Attachments: HDFS-10757.00.patch, HDFS-10757.01.patch, > HDFS-10757.02.patch > > > ClientContext::get gets the context from CACHE via a config setting based > name, then KeyProviderCache stored in ClientContext gets the key provider > cached by URI from the configuration, too. These would return the same > KeyProvider regardless of current UGI. > KMSClientProvider caches the UGI (actualUgi) in ctor; that means in > particular that all the users of DFS with KMSClientProvider in a process will > get the KMS token (along with other credentials) of the first user, via the > above cache. > Either KMSClientProvider shouldn't store the UGI, or one of the caches should > be UGI-aware, like the FS object cache. > Side note: the comment in createConnection that purports to handle the > different UGI doesn't seem to cover what it says it covers. In our case, we > have two unrelated UGIs with no auth (createRemoteUser) with bunch of tokens, > including a KMS token, added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10879) TestEncryptionZonesWithKMS#testReadWrite fails intermitently
[
https://issues.apache.org/jira/browse/HDFS-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508137#comment-15508137
]
Hadoop QA commented on HDFS-10879:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 59m
15s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10879 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829449/HDFS-10879.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2698ab06aeab 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e80386d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16813/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16813/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestEncryptionZonesWithKMS#testReadWrite fails intermitently
>
>
> Key: HDFS-10879
> URL: https://issues.apache.org/jira/browse/HDFS-10879
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Labels: test
> Attachments: HDFS-10879.01.patch
>
>
> {noformat}
> Error Message:
> Key was rolled, versions should be different. Actual: test_key@0
> Stack Trace:
> java.lang.AssertionError: Key was rolled, versions should be different.
> Actual: test_key@0
>
[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Attachment: HADOOP-10757.02.patch Thanks [~xiaochen] for the review. Attached a new patch to address the comments. > KMSClientProvider combined with KeyProviderCache can result in wrong UGI > being used > --- > > Key: HDFS-10757 > URL: https://issues.apache.org/jira/browse/HDFS-10757 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Xiaoyu Yao >Priority: Critical > Attachments: HADOOP-10757.02.patch, HDFS-10757.00.patch, > HDFS-10757.01.patch > > > ClientContext::get gets the context from CACHE via a config setting based > name, then KeyProviderCache stored in ClientContext gets the key provider > cached by URI from the configuration, too. These would return the same > KeyProvider regardless of current UGI. > KMSClientProvider caches the UGI (actualUgi) in ctor; that means in > particular that all the users of DFS with KMSClientProvider in a process will > get the KMS token (along with other credentials) of the first user, via the > above cache. > Either KMSClientProvider shouldn't store the UGI, or one of the caches should > be UGI-aware, like the FS object cache. > Side note: the comment in createConnection that purports to handle the > different UGI doesn't seem to cover what it says it covers. In our case, we > have two unrelated UGIs with no auth (createRemoteUser) with bunch of tokens, > including a KMS token, added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10687) Federation Membership State Store internal APIs
[ https://issues.apache.org/jira/browse/HDFS-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Kace updated HDFS-10687: -- Status: Patch Available (was: Open) > Federation Membership State Store internal APIs > --- > > Key: HDFS-10687 > URL: https://issues.apache.org/jira/browse/HDFS-10687 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Inigo Goiri >Assignee: Jason Kace > Attachments: HDFS-10467-HDFS-10687-001.patch > > > The Federation Membership State encapsulates the information about the > Namenodes of each sub-cluster that are participating in Federation. The > information includes addresses for RPC, Web. This information is stored in > the State Store and later used by the Router to find data in the federation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10687) Federation Membership State Store internal APIs
[ https://issues.apache.org/jira/browse/HDFS-10687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Kace updated HDFS-10687: -- Attachment: HDFS-10467-HDFS-10687-001.patch Attaching first patch version for review. This patch contains some overlap code (common dependencies) with HDFS-10630 (Federation State Store). The code compiles as is, but is not functional until HDFS-10630 is committed. The Federation membership state API functions are defined in org.apache.hadoop.hdfs.server.federation.store.FederationMembershipStateStore.java and are implemented in org.apache.hadoop.hdfs.server.federation.store.impl.FederationMembershipStateStoreImpl.java. The primary function of the membership state API is to persist the data required to identify the most active, or recently active, NN for each nameservice. This is a key feature for HDFS federation. Individual routers monitor 1-N namenodes and heartbeat the namenode status into the state store. The NN heartbeat code is available for review in HDFS-10629. This patch contains the APIs to query, update and cache the reported state of all NNs in the cluster. The additional files pertain to request/response API objects and the protobuf backing implementation of them. All comments and feedback are welcome! > Federation Membership State Store internal APIs > --- > > Key: HDFS-10687 > URL: https://issues.apache.org/jira/browse/HDFS-10687 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Inigo Goiri >Assignee: Jason Kace > Attachments: HDFS-10467-HDFS-10687-001.patch > > > The Federation Membership State encapsulates the information about the > Namenodes of each sub-cluster that are participating in Federation. The > information includes addresses for RPC, Web. This information is stored in > the State Store and later used by the Router to find data in the federation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10880) Federation Mount Table State Store internal API
Jason Kace created HDFS-10880: - Summary: Federation Mount Table State Store internal API Key: HDFS-10880 URL: https://issues.apache.org/jira/browse/HDFS-10880 Project: Hadoop HDFS Issue Type: Sub-task Components: fs Reporter: Jason Kace Assignee: Jason Kace The Federation Mount Table State encapsulates the mapping of file paths in the global namespace to a specific NN(nameservice) and local NN path. The mount table is shared by all router instances and represents a unified view of the global namespace. The state store API for the mount table allows the related records to be queried, updated and deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10877) Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers
[ https://issues.apache.org/jira/browse/HDFS-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508101#comment-15508101 ] Andrew Wang commented on HDFS-10877: Thanks for working on this Sean! Patch LGTM, though we should fix that long line that checkstyle picked up. Test failures look unrelated. +1 pending > Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers > -- > > Key: HDFS-10877 > URL: https://issues.apache.org/jira/browse/HDFS-10877 > Project: Hadoop HDFS > Issue Type: Improvement > Components: qjm >Affects Versions: 3.0.0-alpha1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HDFS-10877.001.patch > > > HDFS-10519 introduced a new field in the RemoteEditLogManifest message. It > can be made optional to improve wire-compatibility with previous versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7878) API - expose an unique file identifier
[
https://issues.apache.org/jira/browse/HDFS-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508091#comment-15508091
]
Chris Douglas commented on HDFS-7878:
-
bq. Any changes to FileStatus has to be done so that external filesystems (e.g.
google cloud storage) which subclass FileStatus don't break. I know, given the
pain Guice causes us it'd be retaliation, but the GCS team aren't the guice
team, and would upset users.
Of course. On HDFS-6984, I'm anxious changing the {{FileStatus}} serialization
between 2.x and 3.x. Distcp is fine- that dependency is only intra-job- but if
someone has a {{SequenceFile}} full of {{FileStatus}} objects in their
clusters, those would become unreadable without a 2.x common jar. Whether it's
worth it or not, we can discuss in HDFS-6984.
bq. FileStatus is part of the public FS API, documented in FileSystem.md.
You're proposing changing it, aren't you? Which means you get to update the doc
and the tests in AbstractContractGetFileStatusTest
Happily. Do you have feedback on this proposal, outside of its requirement to
update the FS specification?
> API - expose an unique file identifier
> --
>
> Key: HDFS-7878
> URL: https://issues.apache.org/jira/browse/HDFS-7878
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Labels: BB2015-05-TBR
> Attachments: HDFS-7878.01.patch, HDFS-7878.02.patch,
> HDFS-7878.03.patch, HDFS-7878.04.patch, HDFS-7878.05.patch,
> HDFS-7878.06.patch, HDFS-7878.patch
>
>
> See HDFS-487.
> Even though that is resolved as duplicate, the ID is actually not exposed by
> the JIRA it supposedly duplicates.
> INode ID for the file should be easy to expose; alternatively ID could be
> derived from block IDs, to account for appends...
> This is useful e.g. for cache key by file, to make sure cache stays correct
> when file is overwritten.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10879) TestEncryptionZonesWithKMS#testReadWrite fails intermitently
[
https://issues.apache.org/jira/browse/HDFS-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507967#comment-15507967
]
Andrew Wang commented on HDFS-10879:
+1 LGTM, and yea definitely similar to that previous one we fixed. Thanks for
the fixing this Xiao!
> TestEncryptionZonesWithKMS#testReadWrite fails intermitently
>
>
> Key: HDFS-10879
> URL: https://issues.apache.org/jira/browse/HDFS-10879
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Labels: test
> Attachments: HDFS-10879.01.patch
>
>
> {noformat}
> Error Message:
> Key was rolled, versions should be different. Actual: test_key@0
> Stack Trace:
> java.lang.AssertionError: Key was rolled, versions should be different.
> Actual: test_key@0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failEquals(Assert.java:185)
> at org.junit.Assert.assertNotEquals(Assert.java:161)
> at
> org.apache.hadoop.hdfs.TestEncryptionZones.testReadWrite(TestEncryptionZones.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10630) Federation State Store
[ https://issues.apache.org/jira/browse/HDFS-10630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Kace updated HDFS-10630: -- Status: Patch Available (was: In Progress) First version of Federation State Store base code is ready for patch feedback. > Federation State Store > -- > > Key: HDFS-10630 > URL: https://issues.apache.org/jira/browse/HDFS-10630 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Inigo Goiri >Assignee: Jason Kace > Attachments: HDFS-10630.001.patch, HDFS-10630.002.patch > > > Interface to store the federation shared state across Routers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10630) Federation State Store
[ https://issues.apache.org/jira/browse/HDFS-10630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Kace updated HDFS-10630: -- Attachment: HDFS-10630.002.patch Updating HDFS Federation state store patch. 1) Renaming elements to better resemble yarn federation. 2) Separating dependencies between the state store API interfaces and the root state store code for easier patch review. 3) A few extra comments and fixes. > Federation State Store > -- > > Key: HDFS-10630 > URL: https://issues.apache.org/jira/browse/HDFS-10630 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Inigo Goiri >Assignee: Jason Kace > Attachments: HDFS-10630.001.patch, HDFS-10630.002.patch > > > Interface to store the federation shared state across Routers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10879) TestEncryptionZonesWithKMS#testReadWrite fails intermitently
[
https://issues.apache.org/jira/browse/HDFS-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507849#comment-15507849
]
Xiao Chen commented on HDFS-10879:
--
In my test run, it failed 2/300 before the patch and 0/300 after.
[~andrew.wang], could you take a look? Thanks!
> TestEncryptionZonesWithKMS#testReadWrite fails intermitently
>
>
> Key: HDFS-10879
> URL: https://issues.apache.org/jira/browse/HDFS-10879
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Labels: test
> Attachments: HDFS-10879.01.patch
>
>
> {noformat}
> Error Message:
> Key was rolled, versions should be different. Actual: test_key@0
> Stack Trace:
> java.lang.AssertionError: Key was rolled, versions should be different.
> Actual: test_key@0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failEquals(Assert.java:185)
> at org.junit.Assert.assertNotEquals(Assert.java:161)
> at
> org.apache.hadoop.hdfs.TestEncryptionZones.testReadWrite(TestEncryptionZones.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10878) TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HDFS-10878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507843#comment-15507843
]
Hadoop QA commented on HDFS-10878:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 106 unchanged - 0 fixed = 107 total (was 106) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10878 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829430/HDFS-10878.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3e87bbead7a2 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9f03b40 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16812/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16812/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16812/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16812/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws
> ConcurrentModificationException
> ---
>
[jira] [Updated] (HDFS-10879) TestEncryptionZonesWithKMS#testReadWrite fails intermitently
[
https://issues.apache.org/jira/browse/HDFS-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-10879:
-
Status: Patch Available (was: Open)
> TestEncryptionZonesWithKMS#testReadWrite fails intermitently
>
>
> Key: HDFS-10879
> URL: https://issues.apache.org/jira/browse/HDFS-10879
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-10879.01.patch
>
>
> {noformat}
> Error Message:
> Key was rolled, versions should be different. Actual: test_key@0
> Stack Trace:
> java.lang.AssertionError: Key was rolled, versions should be different.
> Actual: test_key@0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failEquals(Assert.java:185)
> at org.junit.Assert.assertNotEquals(Assert.java:161)
> at
> org.apache.hadoop.hdfs.TestEncryptionZones.testReadWrite(TestEncryptionZones.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10879) TestEncryptionZonesWithKMS#testReadWrite fails intermitently
[
https://issues.apache.org/jira/browse/HDFS-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-10879:
-
Labels: test (was: )
> TestEncryptionZonesWithKMS#testReadWrite fails intermitently
>
>
> Key: HDFS-10879
> URL: https://issues.apache.org/jira/browse/HDFS-10879
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Labels: test
> Attachments: HDFS-10879.01.patch
>
>
> {noformat}
> Error Message:
> Key was rolled, versions should be different. Actual: test_key@0
> Stack Trace:
> java.lang.AssertionError: Key was rolled, versions should be different.
> Actual: test_key@0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failEquals(Assert.java:185)
> at org.junit.Assert.assertNotEquals(Assert.java:161)
> at
> org.apache.hadoop.hdfs.TestEncryptionZones.testReadWrite(TestEncryptionZones.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10879) TestEncryptionZonesWithKMS#testReadWrite fails intermitently
[
https://issues.apache.org/jira/browse/HDFS-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-10879:
-
Attachment: HDFS-10879.01.patch
This is stack trace reminds me of HADOOP-12699. The conclusion there was this
is by design, so improve the test similarly here.
Had a quick search in the code, didn't find any other places checking this.
> TestEncryptionZonesWithKMS#testReadWrite fails intermitently
>
>
> Key: HDFS-10879
> URL: https://issues.apache.org/jira/browse/HDFS-10879
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Labels: test
> Attachments: HDFS-10879.01.patch
>
>
> {noformat}
> Error Message:
> Key was rolled, versions should be different. Actual: test_key@0
> Stack Trace:
> java.lang.AssertionError: Key was rolled, versions should be different.
> Actual: test_key@0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failEquals(Assert.java:185)
> at org.junit.Assert.assertNotEquals(Assert.java:161)
> at
> org.apache.hadoop.hdfs.TestEncryptionZones.testReadWrite(TestEncryptionZones.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9688) Test the effect of nested encryption zones in HDFS downgrade
[ https://issues.apache.org/jira/browse/HDFS-9688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507817#comment-15507817 ] Chris Trezzo commented on HDFS-9688: Thanks for the pointer to HDFS-7530 [~xyao]! > Test the effect of nested encryption zones in HDFS downgrade > > > Key: HDFS-9688 > URL: https://issues.apache.org/jira/browse/HDFS-9688 > Project: Hadoop HDFS > Issue Type: Test > Components: encryption, test >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: 2.7.3, 2.6.5 > > Attachments: HDFS-9688-branch-2.6.00.patch, > HDFS-9688-branch-2.6.01.patch, HDFS-9688-branch-2.7.03.patch, > HDFS-9688-branch-2.7.04.patch, HDFS-9688-branch-2.8.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10879) TestEncryptionZonesWithKMS#testReadWrite fails intermitently
Xiao Chen created HDFS-10879:
Summary: TestEncryptionZonesWithKMS#testReadWrite fails
intermitently
Key: HDFS-10879
URL: https://issues.apache.org/jira/browse/HDFS-10879
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Xiao Chen
Assignee: Xiao Chen
{noformat}
Error Message:
Key was rolled, versions should be different. Actual: test_key@0
Stack Trace:
java.lang.AssertionError: Key was rolled, versions should be different. Actual:
test_key@0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failEquals(Assert.java:185)
at org.junit.Assert.assertNotEquals(Assert.java:161)
at
org.apache.hadoop.hdfs.TestEncryptionZones.testReadWrite(TestEncryptionZones.java:726)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10800) [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks which were placed in other storages than what NN is expecting.
[ https://issues.apache.org/jira/browse/HDFS-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-10800: --- Attachment: HDFS-10800-HDFS-10285-03.patch Thank you [~rakeshr] for the review. I have updated the patch fixing all your comments. for #4) I have left as is. We can refine later after adding dynamic start/stop option. TODO was there about the same. Thanks Please review! > [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks > which were placed in other storages than what NN is expecting. > --- > > Key: HDFS-10800 > URL: https://issues.apache.org/jira/browse/HDFS-10800 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10800-HDFS-10285-00.patch, > HDFS-10800-HDFS-10285-01.patch, HDFS-10800-HDFS-10285-02.patch, > HDFS-10800-HDFS-10285-03.patch > > > This JIRA is for implementing a daemon thread called StoragePolicySatisfier > in nematode, which should scan the asked files blocks which were placed in > wrong storages in DNs. > The idea is: > # When user called on some files/dirs for satisfyStorage policy, They > should have tracked in NN and then StoragePolicyDaemon thread will pick one > by one file and then check the blocks which might have placed in wrong > storage in DN than what NN is expecting it to. > # After checking all, it should also construct the data structures for > the required information to move a block from one storage to another. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-10872 started by Erik Krogen. -- > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10314) A new tool to sync current HDFS view to specified snapshot
[
https://issues.apache.org/jira/browse/HDFS-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507688#comment-15507688
]
Jing Zhao commented on HDFS-10314:
--
So the current patch actually adds a new distsync extension and implements the
"calculating diff on target cluster" approach? I think to have the diff
calculated on target is fine, but I'm not sure to directly extend the current
distcp is a good idea.
Correct me if I'm wrong. Here's my current understanding of the patch:
1. our main motivation is still to utilize distcp to restore a snapshot
2. the idea is to compute the delta on the target cluster, and for modified
files we get their original state from the source.
In that sense, I think a simpler way is to wrap (but not extend) the current
distcp in the snapshot-restore tool:
1. The tool takes a single cluster and a target snapshot as arguments
2. The tool computes the delta for restoring using snapshot diff report
3. The tool does rename/delete etc. metadata ops to revert part of the diff
4. The tool uses the distcp (by invokes distcp as a library) to copy the
original states of modified files
In this way we can minimize the change (no need to touch the current distcp
implementation/arguments), and provides a new tool with simple but clear
semantic. We may lose some flexibility (only handling one cluster) but the tool
itself will be easy to use and will not cause any confusion to the end users.
What do you think? Please let me know if I miss anything.
> A new tool to sync current HDFS view to specified snapshot
> --
>
> Key: HDFS-10314
> URL: https://issues.apache.org/jira/browse/HDFS-10314
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-10314.001.patch
>
>
> HDFS-9820 proposed adding -rdiff switch to distcp, as a reversed operation of
> -diff switch.
> Upon discussion with [~jingzhao], we will introduce a new tool that wraps
> around distcp to achieve the same purpose.
> I'm thinking about calling the new tool "rsync", similar to unix/linux
> command "rsync". The "r" here means remote.
> The syntax that simulate -rdiff behavior proposed in HDFS-9820 is
> {code}
> rsync
> {code}
> This command ensure is newer than .
> I think, In the future, we can add another command to have the functionality
> of -diff switch of distcp.
> {code}
> sync
> {code}
> that ensures is older than .
> Thanks [~jingzhao].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9688) Test the effect of nested encryption zones in HDFS downgrade
[ https://issues.apache.org/jira/browse/HDFS-9688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507691#comment-15507691 ] Xiaoyu Yao commented on HDFS-9688: -- [~ctrezzo], move the the root of encryption zone is allowed from Hadoop 2.7 by HDFS-7530 but not Hadoop 2.6. The test should have passed if you backport HDFS-7530 to branch-2.6.5. > Test the effect of nested encryption zones in HDFS downgrade > > > Key: HDFS-9688 > URL: https://issues.apache.org/jira/browse/HDFS-9688 > Project: Hadoop HDFS > Issue Type: Test > Components: encryption, test >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: 2.7.3, 2.6.5 > > Attachments: HDFS-9688-branch-2.6.00.patch, > HDFS-9688-branch-2.6.01.patch, HDFS-9688-branch-2.7.03.patch, > HDFS-9688-branch-2.7.04.patch, HDFS-9688-branch-2.8.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10877) Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers
[
https://issues.apache.org/jira/browse/HDFS-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507680#comment-15507680
]
Hadoop QA commented on HDFS-10877:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 56 unchanged - 0 fixed = 57 total (was 56) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.TestErasureCodeBenchmarkThroughput |
| | hadoop.hdfs.TestBlockStoragePolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10877 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829420/HDFS-10877.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 9f6b73cf3914 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e45307c |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16811/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16811/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16811/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16811/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT
[jira] [Commented] (HDFS-10824) MiniDFSCluster#storageCapacities has no effects on real capacity
[ https://issues.apache.org/jira/browse/HDFS-10824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507671#comment-15507671 ] Xiaobing Zhou commented on HDFS-10824: -- Posted v004. Waiting for data node fully started is correct way for syncing storage initialization. > MiniDFSCluster#storageCapacities has no effects on real capacity > > > Key: HDFS-10824 > URL: https://issues.apache.org/jira/browse/HDFS-10824 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10824.000.patch, HDFS-10824.001.patch, > HDFS-10824.002.patch, HDFS-10824.003.patch, HDFS-10824.004.patch > > > It has been noticed MiniDFSCluster#storageCapacities has no effects on real > capacity. It can be reproduced by explicitly setting storageCapacities and > then call ClientProtocol#getDatanodeStorageReport(DatanodeReportType.LIVE) to > compare results. The following are storage report for one node with two > volumes after I set capacity as 300 * 1024. Apparently, the capacity is not > changed. > adminState|DatanodeInfo$AdminStates (id=6861) > |blockPoolUsed|215192| > |cacheCapacity|0| > |cacheUsed|0| > |capacity|998164971520| > |datanodeUuid|"839912e9-5bcb-45d1-81cf-9a9c9c02a00b" (id=6862)| > |dependentHostNames|LinkedList (id=6863)| > |dfsUsed|215192| > |hostName|"127.0.0.1" (id=6864)| > |infoPort|64222| > |infoSecurePort|0| > |ipAddr|"127.0.0.1" (id=6865)| > |ipcPort|64223| > |lastUpdate|1472682790948| > |lastUpdateMonotonic|209605640| > |level|0| > |location|"/default-rack" (id=6866)| > |maintenanceExpireTimeInMS|0| > |parent|null| > |peerHostName|null| > |remaining|20486512640| > |softwareVersion|null| > |upgradeDomain|null| > |xceiverCount|1| > |xferAddr|"127.0.0.1:64220" (id=6855)| > |xferPort|64220| > [0]StorageReport (id=6856) > |blockPoolUsed|4096| > |capacity|499082485760| > |dfsUsed|4096| > |failed|false| > |remaining|10243256320| > |storage|DatanodeStorage (id=6869)| > [1]StorageReport (id=6859) > |blockPoolUsed|211096| > |capacity|499082485760| > |dfsUsed|211096| > |failed|false| > |remaining|10243256320| > |storage|DatanodeStorage (id=6872)| -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10824) MiniDFSCluster#storageCapacities has no effects on real capacity
[ https://issues.apache.org/jira/browse/HDFS-10824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10824: - Attachment: HDFS-10824.004.patch > MiniDFSCluster#storageCapacities has no effects on real capacity > > > Key: HDFS-10824 > URL: https://issues.apache.org/jira/browse/HDFS-10824 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10824.000.patch, HDFS-10824.001.patch, > HDFS-10824.002.patch, HDFS-10824.003.patch, HDFS-10824.004.patch > > > It has been noticed MiniDFSCluster#storageCapacities has no effects on real > capacity. It can be reproduced by explicitly setting storageCapacities and > then call ClientProtocol#getDatanodeStorageReport(DatanodeReportType.LIVE) to > compare results. The following are storage report for one node with two > volumes after I set capacity as 300 * 1024. Apparently, the capacity is not > changed. > adminState|DatanodeInfo$AdminStates (id=6861) > |blockPoolUsed|215192| > |cacheCapacity|0| > |cacheUsed|0| > |capacity|998164971520| > |datanodeUuid|"839912e9-5bcb-45d1-81cf-9a9c9c02a00b" (id=6862)| > |dependentHostNames|LinkedList (id=6863)| > |dfsUsed|215192| > |hostName|"127.0.0.1" (id=6864)| > |infoPort|64222| > |infoSecurePort|0| > |ipAddr|"127.0.0.1" (id=6865)| > |ipcPort|64223| > |lastUpdate|1472682790948| > |lastUpdateMonotonic|209605640| > |level|0| > |location|"/default-rack" (id=6866)| > |maintenanceExpireTimeInMS|0| > |parent|null| > |peerHostName|null| > |remaining|20486512640| > |softwareVersion|null| > |upgradeDomain|null| > |xceiverCount|1| > |xferAddr|"127.0.0.1:64220" (id=6855)| > |xferPort|64220| > [0]StorageReport (id=6856) > |blockPoolUsed|4096| > |capacity|499082485760| > |dfsUsed|4096| > |failed|false| > |remaining|10243256320| > |storage|DatanodeStorage (id=6869)| > [1]StorageReport (id=6859) > |blockPoolUsed|211096| > |capacity|499082485760| > |dfsUsed|211096| > |failed|false| > |remaining|10243256320| > |storage|DatanodeStorage (id=6872)| -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6984) In Hadoop 3, make FileStatus serialize itself via protobuf
[
https://issues.apache.org/jira/browse/HDFS-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507650#comment-15507650
]
Steve Loughran commented on HDFS-6984:
--
FWIW, I'd like FileStatus in Hadoop 2 to implement Serializable. Why? makes it
trivial to pass across processes in Spark and similar remote-closure APIs
> In Hadoop 3, make FileStatus serialize itself via protobuf
> --
>
> Key: HDFS-6984
> URL: https://issues.apache.org/jira/browse/HDFS-6984
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Labels: BB2015-05-TBR
> Attachments: HDFS-6984.001.patch, HDFS-6984.002.patch
>
>
> FileStatus was a Writable in Hadoop 2 and earlier. Originally, we used this
> to serialize it and send it over the wire. But in Hadoop 2 and later, we
> have the protobuf {{HdfsFileStatusProto}} which serves to serialize this
> information. The protobuf form is preferable, since it allows us to add new
> fields in a backwards-compatible way. Another issue is that already a lot of
> subclasses of FileStatus don't override the Writable methods of the
> superclass, breaking the interface contract that read(status.write) should be
> equal to the original status.
> In Hadoop 3, we should just make FileStatus serialize itself via protobuf so
> that we don't have to deal with these issues. It's probably too late to do
> this in Hadoop 2, since user code may be relying on the existing FileStatus
> serialization there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10876) Dispatcher#dispatch should log IOException stacktrace
[
https://issues.apache.org/jira/browse/HDFS-10876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507649#comment-15507649
]
Hadoop QA commented on HDFS-10876:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 66m
15s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10876 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829424/HDFS-10876.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4285ffe72eff 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e45307c |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16810/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16810/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Dispatcher#dispatch should log IOException stacktrace
> -
>
> Key: HDFS-10876
> URL: https://issues.apache.org/jira/browse/HDFS-10876
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Trivial
> Labels: newbie
>
[jira] [Commented] (HDFS-7878) API - expose an unique file identifier
[
https://issues.apache.org/jira/browse/HDFS-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507647#comment-15507647
]
Steve Loughran commented on HDFS-7878:
--
hmmm.
* S3a adds an {{S3AFileStatus}} with a new field "isEmptyDir". I've
contemplated making it a subclass of {{LocatedFileStatus}}. so lists operations
returning those wouldn't need any new objects.
* grepping the Hadoop production code, FileStatus.getPath is often used as the
input to FS operations (open, delete, rename). But not so often for create(),
and rename destinations.
* Any changes to FileStatus has to be done so that external filesystems (e.g.
google cloud storage) which subclass FileStatus don't break. I know, given the
pain Guice causes us it'd be retaliation, but the GCS team aren't the guice
team, and would upset users.
Now, the bad news. FileStatus is part of the public FS API, documented in
{{FileSystem.md}}. You're proposing changing it, aren't you?
Which means you get to update the doc and the tests in
{{AbstractContractGetFileStatusTest}}. And now that I know that JIRA is
planning to change the file, I'm going to be expecting to see that handles.
> API - expose an unique file identifier
> --
>
> Key: HDFS-7878
> URL: https://issues.apache.org/jira/browse/HDFS-7878
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Labels: BB2015-05-TBR
> Attachments: HDFS-7878.01.patch, HDFS-7878.02.patch,
> HDFS-7878.03.patch, HDFS-7878.04.patch, HDFS-7878.05.patch,
> HDFS-7878.06.patch, HDFS-7878.patch
>
>
> See HDFS-487.
> Even though that is resolved as duplicate, the ID is actually not exposed by
> the JIRA it supposedly duplicates.
> INode ID for the file should be easy to expose; alternatively ID could be
> derived from block IDs, to account for appends...
> This is useful e.g. for cache key by file, to make sure cache stays correct
> when file is overwritten.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-9688) Test the effect of nested encryption zones in HDFS downgrade
[
https://issues.apache.org/jira/browse/HDFS-9688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507419#comment-15507419
]
Chris Trezzo edited comment on HDFS-9688 at 9/20/16 7:56 PM:
-
Hi [~zhz] [~andrew.wang]. TestNestedEncryptionZones#testNestedEncryptionZones
seems to be failing in
[branch-2.6.5|https://builds.apache.org/job/Hadoop-branch2-parameterized/13/testReport/org.apache.hadoop.hdfs.server.namenode/TestNestedEncryptionZones/testNestedEncryptionZones/]
with the following error message:
{noformat}
java.lang.AssertionError: Should be able to rename the root dir of an EZ.
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.hdfs.server.namenode.TestNestedEncryptionZones.testNestedEncryptionZones(TestNestedEncryptionZones.java:194)
{noformat}
{noformat}
2016-09-16 23:09:42,930 INFO ipc.Server (Server.java:run(2061)) - IPC Server
handler 9 on 60066, call org.apache.hadoop.hdfs.protocol.ClientProtocol.rename
from 127.0.0.1:48977 Call#195 Retry#0
java.io.IOException: /topEZ/renamedFile can't be moved from encryption zone
/topEZ to encryption zone /topEZ/nestedEZ.
at
org.apache.hadoop.hdfs.server.namenode.EncryptionZoneManager.checkMoveValidity(EncryptionZoneManager.java:285)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedRenameTo(FSDirectory.java:563)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.renameTo(FSDirectory.java:477)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameToInternal(FSNamesystem.java:3857)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameToInt(FSNamesystem.java:3819)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3784)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:787)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:565)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:975)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034)
{noformat}
I can reproduce this on my local machine as well. I am currently investigating
this. Does this failure ring any bells? I also see that this passes on
branch-2.7 and branch-2.7.3. Thanks!
was (Author: ctrezzo):
Hi [~zhz] [~andrew.wang]. TestNestedEncryptionZones#testNestedEncryptionZones
seems to be failing in
[branch-2.6.5|https://builds.apache.org/job/Hadoop-branch2-parameterized/13/testReport/org.apache.hadoop.hdfs.server.namenode/TestNestedEncryptionZones/testNestedEncryptionZones/]
with the following error message:
{noformat}
java.lang.AssertionError: Should be able to rename the root dir of an EZ.
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.hdfs.server.namenode.TestNestedEncryptionZones.testNestedEncryptionZones(TestNestedEncryptionZones.java:194)
{noformat}
{noformat}
2016-09-16 23:09:42,930 INFO ipc.Server (Server.java:run(2061)) - IPC Server
handler 9 on 60066, call org.apache.hadoop.hdfs.protocol.ClientProtocol.rename
from 127.0.0.1:48977 Call#195 Retry#0
java.io.IOException: /topEZ/renamedFile can't be moved from encryption zone
/topEZ to encryption zone /topEZ/nestedEZ.
at
org.apache.hadoop.hdfs.server.namenode.EncryptionZoneManager.checkMoveValidity(EncryptionZoneManager.java:285)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedRenameTo(FSDirectory.java:563)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.renameTo(FSDirectory.java:477)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameToInternal(FSNamesystem.java:3857)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameToInt(FSNamesystem.java:3819)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3784)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:787)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:565)
at
[jira] [Commented] (HDFS-9333) Some tests using MiniDFSCluster errored complaining port in use
[
https://issues.apache.org/jira/browse/HDFS-9333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507620#comment-15507620
]
Andrew Wang commented on HDFS-9333:
---
LGTM +1, test failures look unrelated. Thanks for the working on this
[~iwasakims]!
> Some tests using MiniDFSCluster errored complaining port in use
> ---
>
> Key: HDFS-9333
> URL: https://issues.apache.org/jira/browse/HDFS-9333
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Reporter: Kai Zheng
>Assignee: Masatake Iwasaki
>Priority: Minor
> Attachments: HDFS-9333.001.patch, HDFS-9333.002.patch,
> HDFS-9333.003.patch
>
>
> Ref. the following:
> {noformat}
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.483 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
> testRead(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped)
> Time elapsed: 11.021 sec <<< ERROR!
> java.net.BindException: Port in use: localhost:49333
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> at
> org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:884)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:826)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:821)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:675)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2015)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1996)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS.doTestRead(TestBlockTokenWithDFS.java:539)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped.testRead(TestBlockTokenWithDFSStriped.java:62)
> {noformat}
> Another one:
> {noformat}
> Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 9.859 sec <<<
> FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController
> testFailoverAndBackOnNNShutdown(org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController)
> Time elapsed: 0.41 sec <<< ERROR!
> java.net.BindException: Problem binding to [localhost:10021]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:399)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1245)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1014)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:889)
> at
>
[jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
[ https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507610#comment-15507610 ] Konstantin Shvachko commented on HDFS-10301: Besides, what is the last report? Reports can come to the NameNode in arbitrary order, can get lost, or can be repeated. Looks like this is the last remaining issue. Can we please resolve it soon. > BlockReport retransmissions may lead to storages falsely being declared > zombie if storage report processing happens out of order > > > Key: HDFS-10301 > URL: https://issues.apache.org/jira/browse/HDFS-10301 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.1 >Reporter: Konstantin Shvachko >Assignee: Vinitha Reddy Gankidi >Priority: Critical > Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, > HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.006.patch, > HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch, > HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch, > HDFS-10301.012.patch, HDFS-10301.013.patch, HDFS-10301.014.patch, > HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch, > HDFS-10301.sample.patch, zombieStorageLogs.rtf > > > When NameNode is busy a DataNode can timeout sending a block report. Then it > sends the block report again. Then NameNode while process these two reports > at the same time can interleave processing storages from different reports. > This screws up the blockReportId field, which makes NameNode think that some > storages are zombie. Replicas from zombie storages are immediately removed, > causing missing blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10878) TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HDFS-10878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-10878:
--
Attachment: HDFS-10878.patch
Here is a simple patch which removes the invocation arguments from the log line.
There is no need to print any ways since there is only one block.
> TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws
> ConcurrentModificationException
> ---
>
> Key: HDFS-10878
> URL: https://issues.apache.org/jira/browse/HDFS-10878
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.3
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HDFS-10878.patch
>
>
> This failed in our internal build
> {noformat}
> java.util.ConcurrentModificationException: null
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> at java.util.ArrayList$Itr.next(ArrayList.java:851)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.appendUCParts(BlockInfoContiguousUnderConstruction.java:396)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.appendStringTo(BlockInfoContiguousUnderConstruction.java:382)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.toString(BlockInfoContiguousUnderConstruction.java:375)
> at java.lang.String.valueOf(String.java:2982)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at
> org.apache.hadoop.hdfs.protocol.ExtendedBlock.toString(ExtendedBlock.java:121)
> at com.google.common.base.Joiner.toString(Joiner.java:533)
> at com.google.common.base.Joiner.appendTo(Joiner.java:124)
> at com.google.common.base.Joiner.appendTo(Joiner.java:181)
> at com.google.common.base.Joiner.join(Joiner.java:237)
> at com.google.common.base.Joiner.join(Joiner.java:226)
> at com.google.common.base.Joiner.join(Joiner.java:245)
> at
> org.apache.hadoop.hdfs.TestDFSClientRetries$3.answer(TestDFSClientRetries.java:485)
> at
> org.apache.hadoop.hdfs.TestDFSClientRetries$3.answer(TestDFSClientRetries.java:477)
> at
> org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:31)
> at org.mockito.internal.MockHandler.handle(MockHandler.java:97)
> at
> org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer$$EnhancerByMockitoWithCGLIB$$cca97ed1.complete()
> at
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2303)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2279)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2243)
> at
> org.apache.hadoop.hdfs.TestDFSClientRetries.testIdempotentAllocateBlockAndClose(TestDFSClientRetries.java:507)
> {noformat}
> Its getting NPE in following Log message
> {code:title=TestDFSClientRetries.java|borderStyle=solid}
> @Test
> public void testIdempotentAllocateBlockAndClose() throws Exception {
> ...
> public Boolean answer(InvocationOnMock invocation) throws
> Throwable {
> // complete() may return false a few times before it returns
> // true. We want to wait until it returns true, and then
> // make it retry one more time after that.
> LOG.info("Called complete(: " +
> Joiner.on(",").join(invocation.getArguments()) + ")");
>...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10878) TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HDFS-10878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-10878:
--
Status: Patch Available (was: Open)
> TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws
> ConcurrentModificationException
> ---
>
> Key: HDFS-10878
> URL: https://issues.apache.org/jira/browse/HDFS-10878
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.3
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HDFS-10878.patch
>
>
> This failed in our internal build
> {noformat}
> java.util.ConcurrentModificationException: null
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> at java.util.ArrayList$Itr.next(ArrayList.java:851)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.appendUCParts(BlockInfoContiguousUnderConstruction.java:396)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.appendStringTo(BlockInfoContiguousUnderConstruction.java:382)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.toString(BlockInfoContiguousUnderConstruction.java:375)
> at java.lang.String.valueOf(String.java:2982)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at
> org.apache.hadoop.hdfs.protocol.ExtendedBlock.toString(ExtendedBlock.java:121)
> at com.google.common.base.Joiner.toString(Joiner.java:533)
> at com.google.common.base.Joiner.appendTo(Joiner.java:124)
> at com.google.common.base.Joiner.appendTo(Joiner.java:181)
> at com.google.common.base.Joiner.join(Joiner.java:237)
> at com.google.common.base.Joiner.join(Joiner.java:226)
> at com.google.common.base.Joiner.join(Joiner.java:245)
> at
> org.apache.hadoop.hdfs.TestDFSClientRetries$3.answer(TestDFSClientRetries.java:485)
> at
> org.apache.hadoop.hdfs.TestDFSClientRetries$3.answer(TestDFSClientRetries.java:477)
> at
> org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:31)
> at org.mockito.internal.MockHandler.handle(MockHandler.java:97)
> at
> org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer$$EnhancerByMockitoWithCGLIB$$cca97ed1.complete()
> at
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2303)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2279)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2243)
> at
> org.apache.hadoop.hdfs.TestDFSClientRetries.testIdempotentAllocateBlockAndClose(TestDFSClientRetries.java:507)
> {noformat}
> Its getting NPE in following Log message
> {code:title=TestDFSClientRetries.java|borderStyle=solid}
> @Test
> public void testIdempotentAllocateBlockAndClose() throws Exception {
> ...
> public Boolean answer(InvocationOnMock invocation) throws
> Throwable {
> // complete() may return false a few times before it returns
> // true. We want to wait until it returns true, and then
> // make it retry one more time after that.
> LOG.info("Called complete(: " +
> Joiner.on(",").join(invocation.getArguments()) + ")");
>...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10876) Dispatcher#dispatch should log IOException stacktrace
[
https://issues.apache.org/jira/browse/HDFS-10876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507539#comment-15507539
]
Wei-Chiu Chuang commented on HDFS-10876:
INFO may be fine.
However, I am seeing an IOE in balancer that disrupts normal balancer
operation. It only occurs after several hours of balancing.
2016-05-14 08:56:27,636 WARN [pool-2302-thread-7] balancer.Dispatcher
(Dispatcher.java:dispatch(325)) - Failed to move blk_1550828018_1100035582559
with size=167 from 10.17.1.1:1004:DISK to 10.17.1.2:1004:DISK through
10.17.1.1:1004: Can't re-compute encryption key for nonce, since the required
block key (keyID=-327392027) doesn't exist. Current key: -327392024
Balancer aborts after five iterations if no blocks are moved. If you make it
DEBUG it's going to be really hard to tell what's going on. From a support
perspective it's going to generated many support tickets.
> Dispatcher#dispatch should log IOException stacktrace
> -
>
> Key: HDFS-10876
> URL: https://issues.apache.org/jira/browse/HDFS-10876
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10876.001.patch
>
>
> This error logging should be improved. A warning log should record the
> stacktrace as well.
> {code:title=Dispatcher#dispatch}
> try {
> ...
> } catch (IOException e) {
> LOG.warn("Failed to move " + this + ": " + e.getMessage());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10876) Dispatcher#dispatch should log IOException stacktrace
[
https://issues.apache.org/jira/browse/HDFS-10876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507476#comment-15507476
]
Manoj Govindassamy commented on HDFS-10876:
---
Thanks for the review. Exception is because of the proxy or target issue during
balancing. Upon these issues, 10 second delay for proxy and target are
activated. So, would logging this stacktrace as an INFO message be OK ? But, I
also see your point of not spamming the logs with stack traces and may be DEBUG
for stack trace sounds like a good idea.
[~jojochuang] any thoughts please ?
> Dispatcher#dispatch should log IOException stacktrace
> -
>
> Key: HDFS-10876
> URL: https://issues.apache.org/jira/browse/HDFS-10876
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10876.001.patch
>
>
> This error logging should be improved. A warning log should record the
> stacktrace as well.
> {code:title=Dispatcher#dispatch}
> try {
> ...
> } catch (IOException e) {
> LOG.warn("Failed to move " + this + ": " + e.getMessage());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10876) Dispatcher#dispatch should log IOException stacktrace
[
https://issues.apache.org/jira/browse/HDFS-10876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507436#comment-15507436
]
Mingliang Liu commented on HDFS-10876:
--
I'm also thinking of logging the exception stack trace in INFO or DEBUG level.
> Dispatcher#dispatch should log IOException stacktrace
> -
>
> Key: HDFS-10876
> URL: https://issues.apache.org/jira/browse/HDFS-10876
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10876.001.patch
>
>
> This error logging should be improved. A warning log should record the
> stacktrace as well.
> {code:title=Dispatcher#dispatch}
> try {
> ...
> } catch (IOException e) {
> LOG.warn("Failed to move " + this + ": " + e.getMessage());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10877) Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers
[ https://issues.apache.org/jira/browse/HDFS-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10877: --- Component/s: qjm > Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers > -- > > Key: HDFS-10877 > URL: https://issues.apache.org/jira/browse/HDFS-10877 > Project: Hadoop HDFS > Issue Type: Improvement > Components: qjm >Affects Versions: 3.0.0-alpha1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HDFS-10877.001.patch > > > HDFS-10519 introduced a new field in the RemoteEditLogManifest message. It > can be made optional to improve wire-compatibility with previous versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10876) Dispatcher#dispatch should log IOException stacktrace
[
https://issues.apache.org/jira/browse/HDFS-10876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507431#comment-15507431
]
Mingliang Liu commented on HDFS-10876:
--
+1 pending on Jenkins.
> Dispatcher#dispatch should log IOException stacktrace
> -
>
> Key: HDFS-10876
> URL: https://issues.apache.org/jira/browse/HDFS-10876
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10876.001.patch
>
>
> This error logging should be improved. A warning log should record the
> stacktrace as well.
> {code:title=Dispatcher#dispatch}
> try {
> ...
> } catch (IOException e) {
> LOG.warn("Failed to move " + this + ": " + e.getMessage());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9688) Test the effect of nested encryption zones in HDFS downgrade
[
https://issues.apache.org/jira/browse/HDFS-9688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507419#comment-15507419
]
Chris Trezzo commented on HDFS-9688:
Hi [~zhz] [~andrew.wang]. TestNestedEncryptionZones#testNestedEncryptionZones
seems to be failing in
[branch-2.6.5|https://builds.apache.org/job/Hadoop-branch2-parameterized/13/testReport/org.apache.hadoop.hdfs.server.namenode/TestNestedEncryptionZones/testNestedEncryptionZones/]
with the following error message:
{noformat}
java.lang.AssertionError: Should be able to rename the root dir of an EZ.
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.hdfs.server.namenode.TestNestedEncryptionZones.testNestedEncryptionZones(TestNestedEncryptionZones.java:194)
{noformat}
{noformat}
2016-09-16 23:09:42,930 INFO ipc.Server (Server.java:run(2061)) - IPC Server
handler 9 on 60066, call org.apache.hadoop.hdfs.protocol.ClientProtocol.rename
from 127.0.0.1:48977 Call#195 Retry#0
java.io.IOException: /topEZ/renamedFile can't be moved from encryption zone
/topEZ to encryption zone /topEZ/nestedEZ.
at
org.apache.hadoop.hdfs.server.namenode.EncryptionZoneManager.checkMoveValidity(EncryptionZoneManager.java:285)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedRenameTo(FSDirectory.java:563)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.renameTo(FSDirectory.java:477)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameToInternal(FSNamesystem.java:3857)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameToInt(FSNamesystem.java:3819)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3784)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:787)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:565)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:975)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034)
{noformat}
I can reproduce this on my local machine as well. I am currently investigating
this. Does this failure ring any bells? I also see that this passes on
branch-2.7, but I haven't checked branch-2.7.3 yet. Do you know if this was
fixed sometime between 2.7.3 and the current branch-2.7? Thanks!
> Test the effect of nested encryption zones in HDFS downgrade
>
>
> Key: HDFS-9688
> URL: https://issues.apache.org/jira/browse/HDFS-9688
> Project: Hadoop HDFS
> Issue Type: Test
> Components: encryption, test
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.7.3, 2.6.5
>
> Attachments: HDFS-9688-branch-2.6.00.patch,
> HDFS-9688-branch-2.6.01.patch, HDFS-9688-branch-2.7.03.patch,
> HDFS-9688-branch-2.7.04.patch, HDFS-9688-branch-2.8.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10877) Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers
[ https://issues.apache.org/jira/browse/HDFS-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10877: --- Affects Version/s: 3.0.0-alpha1 Target Version/s: 3.0.0-alpha2 Status: Patch Available (was: Open) > Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers > -- > > Key: HDFS-10877 > URL: https://issues.apache.org/jira/browse/HDFS-10877 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HDFS-10877.001.patch > > > HDFS-10519 introduced a new field in the RemoteEditLogManifest message. It > can be made optional to improve wire-compatibility with previous versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10876) Dispatcher#dispatch should log IOException stacktrace
[
https://issues.apache.org/jira/browse/HDFS-10876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-10876:
--
Target Version/s: 3.0.0-alpha2
Status: Patch Available (was: Open)
> Dispatcher#dispatch should log IOException stacktrace
> -
>
> Key: HDFS-10876
> URL: https://issues.apache.org/jira/browse/HDFS-10876
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10876.001.patch
>
>
> This error logging should be improved. A warning log should record the
> stacktrace as well.
> {code:title=Dispatcher#dispatch}
> try {
> ...
> } catch (IOException e) {
> LOG.warn("Failed to move " + this + ": " + e.getMessage());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10876) Dispatcher#dispatch should log IOException stacktrace
[
https://issues.apache.org/jira/browse/HDFS-10876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-10876:
--
Attachment: HDFS-10876.001.patch
Attached v01 patch to log the exception stacktrace in Dispatcher#dispatch.
> Dispatcher#dispatch should log IOException stacktrace
> -
>
> Key: HDFS-10876
> URL: https://issues.apache.org/jira/browse/HDFS-10876
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10876.001.patch
>
>
> This error logging should be improved. A warning log should record the
> stacktrace as well.
> {code:title=Dispatcher#dispatch}
> try {
> ...
> } catch (IOException e) {
> LOG.warn("Failed to move " + this + ": " + e.getMessage());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10877) Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers
[ https://issues.apache.org/jira/browse/HDFS-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HDFS-10877: - Attachment: HDFS-10877.001.patch Attaching a patch that modifies the .proto definition and the unit test for it. Every use of this field that I can see can already be INVALID_TXID by default anyway, so I believe this is a safe change. > Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers > -- > > Key: HDFS-10877 > URL: https://issues.apache.org/jira/browse/HDFS-10877 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HDFS-10877.001.patch > > > HDFS-10519 introduced a new field in the RemoteEditLogManifest message. It > can be made optional to improve wire-compatibility with previous versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10878) TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HDFS-10878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-10878:
--
Summary: TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws
ConcurrentModificationException (was:
TestDFSClientRetries#testIdempotentAllocateBlockAndClose throwing
ConcurrentModificationException)
> TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws
> ConcurrentModificationException
> ---
>
> Key: HDFS-10878
> URL: https://issues.apache.org/jira/browse/HDFS-10878
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.3
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>
> This failed in out internal build
> {noformat}
> java.util.ConcurrentModificationException: null
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> at java.util.ArrayList$Itr.next(ArrayList.java:851)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.appendUCParts(BlockInfoContiguousUnderConstruction.java:396)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.appendStringTo(BlockInfoContiguousUnderConstruction.java:382)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.toString(BlockInfoContiguousUnderConstruction.java:375)
> at java.lang.String.valueOf(String.java:2982)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at
> org.apache.hadoop.hdfs.protocol.ExtendedBlock.toString(ExtendedBlock.java:121)
> at com.google.common.base.Joiner.toString(Joiner.java:533)
> at com.google.common.base.Joiner.appendTo(Joiner.java:124)
> at com.google.common.base.Joiner.appendTo(Joiner.java:181)
> at com.google.common.base.Joiner.join(Joiner.java:237)
> at com.google.common.base.Joiner.join(Joiner.java:226)
> at com.google.common.base.Joiner.join(Joiner.java:245)
> at
> org.apache.hadoop.hdfs.TestDFSClientRetries$3.answer(TestDFSClientRetries.java:485)
> at
> org.apache.hadoop.hdfs.TestDFSClientRetries$3.answer(TestDFSClientRetries.java:477)
> at
> org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:31)
> at org.mockito.internal.MockHandler.handle(MockHandler.java:97)
> at
> org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer$$EnhancerByMockitoWithCGLIB$$cca97ed1.complete()
> at
> org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2303)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2279)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2243)
> at
> org.apache.hadoop.hdfs.TestDFSClientRetries.testIdempotentAllocateBlockAndClose(TestDFSClientRetries.java:507)
> {noformat}
> Its getting NPE in following Log message
> {code:title=TestDFSClientRetries.java|borderStyle=solid}
> @Test
> public void testIdempotentAllocateBlockAndClose() throws Exception {
> ...
> public Boolean answer(InvocationOnMock invocation) throws
> Throwable {
> // complete() may return false a few times before it returns
> // true. We want to wait until it returns true, and then
> // make it retry one more time after that.
> LOG.info("Called complete(: " +
> Joiner.on(",").join(invocation.getArguments()) + ")");
>...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10878) TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HDFS-10878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-10878:
--
Description:
This failed in our internal build
{noformat}
java.util.ConcurrentModificationException: null
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
at java.util.ArrayList$Itr.next(ArrayList.java:851)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.appendUCParts(BlockInfoContiguousUnderConstruction.java:396)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.appendStringTo(BlockInfoContiguousUnderConstruction.java:382)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.toString(BlockInfoContiguousUnderConstruction.java:375)
at java.lang.String.valueOf(String.java:2982)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at
org.apache.hadoop.hdfs.protocol.ExtendedBlock.toString(ExtendedBlock.java:121)
at com.google.common.base.Joiner.toString(Joiner.java:533)
at com.google.common.base.Joiner.appendTo(Joiner.java:124)
at com.google.common.base.Joiner.appendTo(Joiner.java:181)
at com.google.common.base.Joiner.join(Joiner.java:237)
at com.google.common.base.Joiner.join(Joiner.java:226)
at com.google.common.base.Joiner.join(Joiner.java:245)
at
org.apache.hadoop.hdfs.TestDFSClientRetries$3.answer(TestDFSClientRetries.java:485)
at
org.apache.hadoop.hdfs.TestDFSClientRetries$3.answer(TestDFSClientRetries.java:477)
at
org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:31)
at org.mockito.internal.MockHandler.handle(MockHandler.java:97)
at
org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer$$EnhancerByMockitoWithCGLIB$$cca97ed1.complete()
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2303)
at
org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2279)
at
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2243)
at
org.apache.hadoop.hdfs.TestDFSClientRetries.testIdempotentAllocateBlockAndClose(TestDFSClientRetries.java:507)
{noformat}
Its getting NPE in following Log message
{code:title=TestDFSClientRetries.java|borderStyle=solid}
@Test
public void testIdempotentAllocateBlockAndClose() throws Exception {
...
public Boolean answer(InvocationOnMock invocation) throws Throwable
{
// complete() may return false a few times before it returns
// true. We want to wait until it returns true, and then
// make it retry one more time after that.
LOG.info("Called complete(: " +
Joiner.on(",").join(invocation.getArguments()) + ")");
...
}
{code}
was:
This failed in out internal build
{noformat}
java.util.ConcurrentModificationException: null
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
at java.util.ArrayList$Itr.next(ArrayList.java:851)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.appendUCParts(BlockInfoContiguousUnderConstruction.java:396)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.appendStringTo(BlockInfoContiguousUnderConstruction.java:382)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.toString(BlockInfoContiguousUnderConstruction.java:375)
at java.lang.String.valueOf(String.java:2982)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at
org.apache.hadoop.hdfs.protocol.ExtendedBlock.toString(ExtendedBlock.java:121)
at com.google.common.base.Joiner.toString(Joiner.java:533)
at com.google.common.base.Joiner.appendTo(Joiner.java:124)
at com.google.common.base.Joiner.appendTo(Joiner.java:181)
at com.google.common.base.Joiner.join(Joiner.java:237)
at com.google.common.base.Joiner.join(Joiner.java:226)
at com.google.common.base.Joiner.join(Joiner.java:245)
at
org.apache.hadoop.hdfs.TestDFSClientRetries$3.answer(TestDFSClientRetries.java:485)
at
org.apache.hadoop.hdfs.TestDFSClientRetries$3.answer(TestDFSClientRetries.java:477)
at
org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:31)
at org.mockito.internal.MockHandler.handle(MockHandler.java:97)
at
org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)
at
[jira] [Created] (HDFS-10878) TestDFSClientRetries#testIdempotentAllocateBlockAndClose throwing ConcurrentModificationException
Rushabh S Shah created HDFS-10878:
-
Summary: TestDFSClientRetries#testIdempotentAllocateBlockAndClose
throwing ConcurrentModificationException
Key: HDFS-10878
URL: https://issues.apache.org/jira/browse/HDFS-10878
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 2.7.3
Reporter: Rushabh S Shah
Assignee: Rushabh S Shah
This failed in out internal build
{noformat}
java.util.ConcurrentModificationException: null
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
at java.util.ArrayList$Itr.next(ArrayList.java:851)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.appendUCParts(BlockInfoContiguousUnderConstruction.java:396)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.appendStringTo(BlockInfoContiguousUnderConstruction.java:382)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction.toString(BlockInfoContiguousUnderConstruction.java:375)
at java.lang.String.valueOf(String.java:2982)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at
org.apache.hadoop.hdfs.protocol.ExtendedBlock.toString(ExtendedBlock.java:121)
at com.google.common.base.Joiner.toString(Joiner.java:533)
at com.google.common.base.Joiner.appendTo(Joiner.java:124)
at com.google.common.base.Joiner.appendTo(Joiner.java:181)
at com.google.common.base.Joiner.join(Joiner.java:237)
at com.google.common.base.Joiner.join(Joiner.java:226)
at com.google.common.base.Joiner.join(Joiner.java:245)
at
org.apache.hadoop.hdfs.TestDFSClientRetries$3.answer(TestDFSClientRetries.java:485)
at
org.apache.hadoop.hdfs.TestDFSClientRetries$3.answer(TestDFSClientRetries.java:477)
at
org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:31)
at org.mockito.internal.MockHandler.handle(MockHandler.java:97)
at
org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer$$EnhancerByMockitoWithCGLIB$$cca97ed1.complete()
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2303)
at
org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2279)
at
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2243)
at
org.apache.hadoop.hdfs.TestDFSClientRetries.testIdempotentAllocateBlockAndClose(TestDFSClientRetries.java:507)
{noformat}
Its getting NPE in following Log message
{code:title=TestDFSClientRetries.java|borderStyle=solid}
@Test
public void testIdempotentAllocateBlockAndClose() throws Exception {
...
public Boolean answer(InvocationOnMock invocation) throws Throwable
{
// complete() may return false a few times before it returns
// true. We want to wait until it returns true, and then
// make it retry one more time after that.
LOG.info("Called complete(: " +
Joiner.on(",").join(invocation.getArguments()) + ")");
...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10876) Dispatcher#dispatch should log IOException stacktrace
[
https://issues.apache.org/jira/browse/HDFS-10876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy reassigned HDFS-10876:
-
Assignee: Manoj Govindassamy
> Dispatcher#dispatch should log IOException stacktrace
> -
>
> Key: HDFS-10876
> URL: https://issues.apache.org/jira/browse/HDFS-10876
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Trivial
> Labels: newbie
>
> This error logging should be improved. A warning log should record the
> stacktrace as well.
> {code:title=Dispatcher#dispatch}
> try {
> ...
> } catch (IOException e) {
> LOG.warn("Failed to move " + this + ": " + e.getMessage());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10870) Wrong dfs.namenode.acls.enabled default in HdfsPermissionsGuide.apt.vm
[
https://issues.apache.org/jira/browse/HDFS-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507283#comment-15507283
]
John Zhuge commented on HDFS-10870:
---
Thanks [~eddyxu] for the review and commit!
Thanks [~raviprak], this fix is to prevent users from wrongly expecting ACLs to
be enabled.
> Wrong dfs.namenode.acls.enabled default in HdfsPermissionsGuide.apt.vm
> --
>
> Key: HDFS-10870
> URL: https://issues.apache.org/jira/browse/HDFS-10870
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Trivial
> Fix For: 2.6.5
>
> Attachments: HDFS-10870-branch-2.6.001.patch
>
>
> Wrong {{dfs.namenode.acls.enabled = true}} in
> {{HdfsPermissionsGuide.apt.vm}}. The default should be {{false}} as correctly
> stated in the description and in {{DFS_NAMENODE_ACLS_ENABLED_DEFAULT}}:
> {code}
> * <<>>
>Set to true to enable support for HDFS ACLs (Access Control Lists). By
>default, ACLs are disabled. When ACLs are disabled, the NameNode
> rejects
>all attempts to set an ACL.
> {code}
> {code:title=DFSConfigKeys.java}
> public static final String DFS_NAMENODE_ACLS_ENABLED_KEY =
> "dfs.namenode.acls.enabled";
> public static final boolean DFS_NAMENODE_ACLS_ENABLED_DEFAULT = false;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-10877) Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers
[ https://issues.apache.org/jira/browse/HDFS-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory moved HADOOP-13629 to HDFS-10877: --- Issue Type: Improvement (was: Bug) Key: HDFS-10877 (was: HADOOP-13629) Project: Hadoop HDFS (was: Hadoop Common) > Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers > -- > > Key: HDFS-10877 > URL: https://issues.apache.org/jira/browse/HDFS-10877 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sean Mackrory >Assignee: Sean Mackrory > > HDFS-10519 introduced a new field in the RemoteEditLogManifest message. It > can be made optional to improve wire-compatibility with previous versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-10870) Wrong dfs.namenode.acls.enabled default in HdfsPermissionsGuide.apt.vm
[
https://issues.apache.org/jira/browse/HDFS-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated HDFS-10870:
Comment: was deleted
(was: Would this not be a backward incompatible change?)
> Wrong dfs.namenode.acls.enabled default in HdfsPermissionsGuide.apt.vm
> --
>
> Key: HDFS-10870
> URL: https://issues.apache.org/jira/browse/HDFS-10870
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Trivial
> Fix For: 2.6.5
>
> Attachments: HDFS-10870-branch-2.6.001.patch
>
>
> Wrong {{dfs.namenode.acls.enabled = true}} in
> {{HdfsPermissionsGuide.apt.vm}}. The default should be {{false}} as correctly
> stated in the description and in {{DFS_NAMENODE_ACLS_ENABLED_DEFAULT}}:
> {code}
> * <<>>
>Set to true to enable support for HDFS ACLs (Access Control Lists). By
>default, ACLs are disabled. When ACLs are disabled, the NameNode
> rejects
>all attempts to set an ACL.
> {code}
> {code:title=DFSConfigKeys.java}
> public static final String DFS_NAMENODE_ACLS_ENABLED_KEY =
> "dfs.namenode.acls.enabled";
> public static final boolean DFS_NAMENODE_ACLS_ENABLED_DEFAULT = false;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10870) Wrong dfs.namenode.acls.enabled default in HdfsPermissionsGuide.apt.vm
[
https://issues.apache.org/jira/browse/HDFS-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507175#comment-15507175
]
Ravi Prakash commented on HDFS-10870:
-
Sorry. Never mind. This is just a change in the doc. Thanks Eddy and John
> Wrong dfs.namenode.acls.enabled default in HdfsPermissionsGuide.apt.vm
> --
>
> Key: HDFS-10870
> URL: https://issues.apache.org/jira/browse/HDFS-10870
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Trivial
> Fix For: 2.6.5
>
> Attachments: HDFS-10870-branch-2.6.001.patch
>
>
> Wrong {{dfs.namenode.acls.enabled = true}} in
> {{HdfsPermissionsGuide.apt.vm}}. The default should be {{false}} as correctly
> stated in the description and in {{DFS_NAMENODE_ACLS_ENABLED_DEFAULT}}:
> {code}
> * <<>>
>Set to true to enable support for HDFS ACLs (Access Control Lists). By
>default, ACLs are disabled. When ACLs are disabled, the NameNode
> rejects
>all attempts to set an ACL.
> {code}
> {code:title=DFSConfigKeys.java}
> public static final String DFS_NAMENODE_ACLS_ENABLED_KEY =
> "dfs.namenode.acls.enabled";
> public static final boolean DFS_NAMENODE_ACLS_ENABLED_DEFAULT = false;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10870) Wrong dfs.namenode.acls.enabled default in HdfsPermissionsGuide.apt.vm
[
https://issues.apache.org/jira/browse/HDFS-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507172#comment-15507172
]
Ravi Prakash commented on HDFS-10870:
-
Would this not be a backward incompatible change?
> Wrong dfs.namenode.acls.enabled default in HdfsPermissionsGuide.apt.vm
> --
>
> Key: HDFS-10870
> URL: https://issues.apache.org/jira/browse/HDFS-10870
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Trivial
> Fix For: 2.6.5
>
> Attachments: HDFS-10870-branch-2.6.001.patch
>
>
> Wrong {{dfs.namenode.acls.enabled = true}} in
> {{HdfsPermissionsGuide.apt.vm}}. The default should be {{false}} as correctly
> stated in the description and in {{DFS_NAMENODE_ACLS_ENABLED_DEFAULT}}:
> {code}
> * <<>>
>Set to true to enable support for HDFS ACLs (Access Control Lists). By
>default, ACLs are disabled. When ACLs are disabled, the NameNode
> rejects
>all attempts to set an ACL.
> {code}
> {code:title=DFSConfigKeys.java}
> public static final String DFS_NAMENODE_ACLS_ENABLED_KEY =
> "dfs.namenode.acls.enabled";
> public static final boolean DFS_NAMENODE_ACLS_ENABLED_DEFAULT = false;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10426) TestPendingInvalidateBlock failed in trunk
[
https://issues.apache.org/jira/browse/HDFS-10426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507013#comment-15507013
]
Hadoop QA commented on HDFS-10426:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 1 unchanged - 1 fixed = 1 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 19s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestCrcCorruption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10426 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829396/HDFS-10426.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0ea7aecdb61b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c6d1d74 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16809/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16809/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16809/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestPendingInvalidateBlock failed in trunk
> --
>
> Key: HDFS-10426
> URL: https://issues.apache.org/jira/browse/HDFS-10426
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Yiqun Lin
>
[jira] [Created] (HDFS-10876) Dispatcher#dispatch should log IOException stacktrace
Wei-Chiu Chuang created HDFS-10876:
--
Summary: Dispatcher#dispatch should log IOException stacktrace
Key: HDFS-10876
URL: https://issues.apache.org/jira/browse/HDFS-10876
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer & mover
Affects Versions: 2.6.0
Reporter: Wei-Chiu Chuang
Priority: Trivial
This error logging should be improved. A warning log should record the
stacktrace as well.
{code:title=Dispatcher#dispatch}
try {
...
} catch (IOException e) {
LOG.warn("Failed to move " + this + ": " + e.getMessage());
...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10874) libhdfs++: Public API headers should not depend on internal implementation
[ https://issues.apache.org/jira/browse/HDFS-10874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506847#comment-15506847 ] Bob Hansen commented on HDFS-10874: --- Perhaps as another task, we should ensure that the tools and examples build using just the public headers. > libhdfs++: Public API headers should not depend on internal implementation > -- > > Key: HDFS-10874 > URL: https://issues.apache.org/jira/browse/HDFS-10874 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10874.HDFS-8707.000.patch > > > Public headers need to do some combination of the following: stop including > parts of the implementation, forward declare bits of the implementation where > absolutely needed, or pull the implementation into include/hdfspp if it's > inseparable. > Example: > If you want to use the C++ API and only stick include/hdfspp in the include > path you'll get an error when you include include/hdfspp/options.h because > that goes and includes common/uri.h. > Related to the work described in HDFS-10787. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10426) TestPendingInvalidateBlock failed in trunk
[
https://issues.apache.org/jira/browse/HDFS-10426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10426:
-
Attachment: HDFS-10426.006.patch
> TestPendingInvalidateBlock failed in trunk
> --
>
> Key: HDFS-10426
> URL: https://issues.apache.org/jira/browse/HDFS-10426
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10426.001.patch, HDFS-10426.002.patch,
> HDFS-10426.003.patch, HDFS-10426.004.patch, HDFS-10426.005.patch,
> HDFS-10426.006.patch
>
>
> The test {{TestPendingInvalidateBlock}} failed sometimes. The stack info:
> {code}
> org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
> testPendingDeletion(org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock)
> Time elapsed: 7.703 sec <<< FAILURE!
> java.lang.AssertionError: expected:<2> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeletion(TestPendingInvalidateBlock.java:92)
> {code}
> It looks that the {{invalidateBlock}} has been removed before we do the check
> {code}
> // restart NN
> cluster.restartNameNode(true);
> dfs.delete(foo, true);
> Assert.assertEquals(0, cluster.getNamesystem().getBlocksTotal());
> Assert.assertEquals(REPLICATION, cluster.getNamesystem()
> .getPendingDeletionBlocks());
> Assert.assertEquals(REPLICATION,
> dfs.getPendingDeletionBlocksCount());
> {code}
> And I look into the related configurations. I found the property
> {{dfs.namenode.replication.interval}} was just set as 1 second in this test.
> And after the delay time of {{dfs.namenode.startup.delay.block.deletion.sec}}
> and the delete operation was slowly, it will cause this case. We can see the
> stack info before, the failed test costs 7.7s more than 5+1 second.
> One way can improve this.
> * Increase the time of {{dfs.namenode.replication.interval}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10426) TestPendingInvalidateBlock failed in trunk
[
https://issues.apache.org/jira/browse/HDFS-10426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506656#comment-15506656
]
Yiqun Lin commented on HDFS-10426:
--
Thanks [~iwasakims] for the comments, attach a new patch to address your
comments. Hope my latest patch can satisfy with you.
> TestPendingInvalidateBlock failed in trunk
> --
>
> Key: HDFS-10426
> URL: https://issues.apache.org/jira/browse/HDFS-10426
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10426.001.patch, HDFS-10426.002.patch,
> HDFS-10426.003.patch, HDFS-10426.004.patch, HDFS-10426.005.patch
>
>
> The test {{TestPendingInvalidateBlock}} failed sometimes. The stack info:
> {code}
> org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
> testPendingDeletion(org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock)
> Time elapsed: 7.703 sec <<< FAILURE!
> java.lang.AssertionError: expected:<2> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeletion(TestPendingInvalidateBlock.java:92)
> {code}
> It looks that the {{invalidateBlock}} has been removed before we do the check
> {code}
> // restart NN
> cluster.restartNameNode(true);
> dfs.delete(foo, true);
> Assert.assertEquals(0, cluster.getNamesystem().getBlocksTotal());
> Assert.assertEquals(REPLICATION, cluster.getNamesystem()
> .getPendingDeletionBlocks());
> Assert.assertEquals(REPLICATION,
> dfs.getPendingDeletionBlocksCount());
> {code}
> And I look into the related configurations. I found the property
> {{dfs.namenode.replication.interval}} was just set as 1 second in this test.
> And after the delay time of {{dfs.namenode.startup.delay.block.deletion.sec}}
> and the delete operation was slowly, it will cause this case. We can see the
> stack info before, the failed test costs 7.7s more than 5+1 second.
> One way can improve this.
> * Increase the time of {{dfs.namenode.replication.interval}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10861) Refactor StripeReaders and use ECChunk version decode API
[ https://issues.apache.org/jira/browse/HDFS-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506620#comment-15506620 ] Hudson commented on HDFS-10861: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10465 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10465/]) HDFS-10861. Refactor StripeReaders and use ECChunk version decode API. (kai.zheng: rev 734d54c1a8950446e68098f62d8964e02ecc2890) * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestStripedBlockUtil.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ElasticByteBufferPool.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECChunk.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StatefulStripeReader.java * (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/CoderUtil.java * (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/PositionStripeReader.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java > Refactor StripeReaders and use ECChunk version decode API > - > > Key: HDFS-10861 > URL: https://issues.apache.org/jira/browse/HDFS-10861 > Project: Hadoop HDFS > Issue Type: Task >Reporter: SammiChen >Assignee: SammiChen > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10861-v1.patch, HDFS-10861-v2.patch, > HDFS-10861-v3.patch, HDFS-10861-v4.patch, HDFS-10861-v5.patch, > HDFS-10861-v6.patch > > > Refactor StatefulStripeReader and PositionStripeReader, use ECChunk version > decode API. After the refactor, it is approaching very near now to the ideal > state desired by next step, employing ErasureCoder API instead of > RawErasureCoder API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10861) Refactor StripeReaders and use ECChunk version decode API
[ https://issues.apache.org/jira/browse/HDFS-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-10861: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 Status: Resolved (was: Patch Available) Committed to trunk. Thanks [~Sammi] for the contribution! > Refactor StripeReaders and use ECChunk version decode API > - > > Key: HDFS-10861 > URL: https://issues.apache.org/jira/browse/HDFS-10861 > Project: Hadoop HDFS > Issue Type: Task >Reporter: SammiChen >Assignee: SammiChen > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10861-v1.patch, HDFS-10861-v2.patch, > HDFS-10861-v3.patch, HDFS-10861-v4.patch, HDFS-10861-v5.patch, > HDFS-10861-v6.patch > > > Refactor StatefulStripeReader and PositionStripeReader, use ECChunk version > decode API. After the refactor, it is approaching very near now to the ideal > state desired by next step, employing ErasureCoder API instead of > RawErasureCoder API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10861) Refactor StripeReaders and use ECChunk version decode API
[ https://issues.apache.org/jira/browse/HDFS-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506548#comment-15506548 ] Kai Zheng commented on HDFS-10861: -- Thanks [~Sammi] for the update! The latest patch LGTM now. +1 and will commit it shortly. > Refactor StripeReaders and use ECChunk version decode API > - > > Key: HDFS-10861 > URL: https://issues.apache.org/jira/browse/HDFS-10861 > Project: Hadoop HDFS > Issue Type: Task >Reporter: SammiChen >Assignee: SammiChen > Attachments: HDFS-10861-v1.patch, HDFS-10861-v2.patch, > HDFS-10861-v3.patch, HDFS-10861-v4.patch, HDFS-10861-v5.patch, > HDFS-10861-v6.patch > > > Refactor StatefulStripeReader and PositionStripeReader, use ECChunk version > decode API. After the refactor, it is approaching very near now to the ideal > state desired by next step, employing ErasureCoder API instead of > RawErasureCoder API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10869) Unused method checkId() in InodeId.java file
[
https://issues.apache.org/jira/browse/HDFS-10869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jagadesh Kiran N updated HDFS-10869:
Issue Type: Improvement (was: Bug)
> Unused method checkId() in InodeId.java file
> -
>
> Key: HDFS-10869
> URL: https://issues.apache.org/jira/browse/HDFS-10869
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Jagadesh Kiran N
>Assignee: Jagadesh Kiran N
>
> The following method in InodeId.java is not used anywhere , We can remove the
> code
> {code}
> public static void checkId(long requestId, INode inode)
> throws FileNotFoundException {
> if (requestId != HdfsConstants.GRANDFATHER_INODE_ID && requestId !=
> inode.getId()) {
> throw new FileNotFoundException(
> "ID mismatch. Request id and saved id: " + requestId + " , "
> + inode.getId() + " for file " + inode.getFullPathName());
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9668) Optimize the locking in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506007#comment-15506007
]
Hadoop QA commented on HDFS-9668:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
57s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 39s{color} | {color:orange} root: The patch generated 1 new + 182 unchanged
- 13 fixed = 183 total (was 195) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
21s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 60m
4s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-9668 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829336/HDFS-9668-12.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux bd90b90b1347 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e52d6e7 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16807/artifact/patchprocess/diff-checkstyle-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16807/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16807/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Optimize the
[jira] [Commented] (HDFS-10745) Directly resolve paths into INodesInPath
[
https://issues.apache.org/jira/browse/HDFS-10745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505988#comment-15505988
]
Hadoop QA commented on HDFS-10745:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
43s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
8s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 5 new + 522 unchanged - 6 fixed = 527 total (was 528) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 7964 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 3m
35s{color} | {color:red} The patch 109 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
35s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 47m 20s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_111. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
18s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}146m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Dead store to src in
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getContentSummary(FSDirectory,
String) At
[jira] [Commented] (HDFS-10861) Refactor StripeReaders and use ECChunk version decode API
[
https://issues.apache.org/jira/browse/HDFS-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505975#comment-15505975
]
Hadoop QA commented on HDFS-10861:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 50s{color}
| {color:red} root generated 1 new + 708 unchanged - 1 fixed = 709 total (was
709) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 27s{color} | {color:orange} root: The patch generated 18 new + 88 unchanged
- 51 fixed = 106 total (was 139) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
50s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
4s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 64m
0s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}121m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10861 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829332/HDFS-10861-v6.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6449481a0d82 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e52d6e7 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/16806/artifact/patchprocess/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16806/artifact/patchprocess/diff-checkstyle-root.txt
|
| Test Results |
[jira] [Commented] (HDFS-10861) Refactor StripeReaders and use ECChunk version decode API
[ https://issues.apache.org/jira/browse/HDFS-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505957#comment-15505957 ] SammiChen commented on HDFS-10861: -- Hadoop QA run the test with the old v5 patch > Refactor StripeReaders and use ECChunk version decode API > - > > Key: HDFS-10861 > URL: https://issues.apache.org/jira/browse/HDFS-10861 > Project: Hadoop HDFS > Issue Type: Task >Reporter: SammiChen >Assignee: SammiChen > Attachments: HDFS-10861-v1.patch, HDFS-10861-v2.patch, > HDFS-10861-v3.patch, HDFS-10861-v4.patch, HDFS-10861-v5.patch, > HDFS-10861-v6.patch > > > Refactor StatefulStripeReader and PositionStripeReader, use ECChunk version > decode API. After the refactor, it is approaching very near now to the ideal > state desired by next step, employing ErasureCoder API instead of > RawErasureCoder API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10861) Refactor StripeReaders and use ECChunk version decode API
[ https://issues.apache.org/jira/browse/HDFS-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-10861: - Attachment: HDFS-10861-v6.patch Resubmit patch v6 to trigger Hadoop QA run the test against v6 > Refactor StripeReaders and use ECChunk version decode API > - > > Key: HDFS-10861 > URL: https://issues.apache.org/jira/browse/HDFS-10861 > Project: Hadoop HDFS > Issue Type: Task >Reporter: SammiChen >Assignee: SammiChen > Attachments: HDFS-10861-v1.patch, HDFS-10861-v2.patch, > HDFS-10861-v3.patch, HDFS-10861-v4.patch, HDFS-10861-v5.patch, > HDFS-10861-v6.patch > > > Refactor StatefulStripeReader and PositionStripeReader, use ECChunk version > decode API. After the refactor, it is approaching very near now to the ideal > state desired by next step, employing ErasureCoder API instead of > RawErasureCoder API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10861) Refactor StripeReaders and use ECChunk version decode API
[ https://issues.apache.org/jira/browse/HDFS-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-10861: - Attachment: (was: HDFS-10861-v6.patch) > Refactor StripeReaders and use ECChunk version decode API > - > > Key: HDFS-10861 > URL: https://issues.apache.org/jira/browse/HDFS-10861 > Project: Hadoop HDFS > Issue Type: Task >Reporter: SammiChen >Assignee: SammiChen > Attachments: HDFS-10861-v1.patch, HDFS-10861-v2.patch, > HDFS-10861-v3.patch, HDFS-10861-v4.patch, HDFS-10861-v5.patch > > > Refactor StatefulStripeReader and PositionStripeReader, use ECChunk version > decode API. After the refactor, it is approaching very near now to the ideal > state desired by next step, employing ErasureCoder API instead of > RawErasureCoder API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10861) Refactor StripeReaders and use ECChunk version decode API
[
https://issues.apache.org/jira/browse/HDFS-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505937#comment-15505937
]
Hadoop QA commented on HDFS-10861:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 47s{color}
| {color:red} root generated 1 new + 707 unchanged - 1 fixed = 708 total (was
708) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 32s{color} | {color:orange} root: The patch generated 19 new + 87 unchanged
- 52 fixed = 106 total (was 139) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 59s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
0s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ipc.TestIPC |
| | hadoop.ipc.TestRPC |
| | hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10861 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829166/HDFS-10861-v5.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9a495465eefd 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e52d6e7 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| javac |
[jira] [Commented] (HDFS-10863) hadoop superusergroup supergroup issue
[ https://issues.apache.org/jira/browse/HDFS-10863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505776#comment-15505776 ] Yulei Li commented on HDFS-10863: - According to your setup, I found my linux user matched hdfs: [hadoop@hadoop ~]$ ll test -rw-rw-r--. 1 hadoop hadoop 5 9月 20 14:25 test [hadoop@hadoop ~]$ hdfs dfs -ls /tmp Found 1 items -rw-r--r-- 1 hadoop hadoop 5 2016-09-20 14:30 /tmp/test My OS is CentOS7, and hadoop version is hadoop 3.0.0, I don't think it is a bug. Could you show some more information about your environment and configuration? > hadoop superusergroup supergroup issue > -- > > Key: HDFS-10863 > URL: https://issues.apache.org/jira/browse/HDFS-10863 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.2 > Environment: $ hadoop version > Hadoop 2.7.2 >Reporter: www.jbigdata.fr >Priority: Minor > > I want to match my unix user to HDFS: hduser:hadoop. > For the user I use the VE. > $ echo $HADOOP_HDFS_USER > hduser > For the group I use the hdfs-site.xml : > > dfs.permissions.superusergroup > hadoop > > The namenode log file show the parameter user/group values. > INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner > = hduser (auth:SIMPLE) > INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup > = hadoop > INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled > = true > Everything seems to be OK, but when I copy file form FS to HDFS the group is > not correct. It keeps the supergroup default value. > Thoses shell commands show the issue: > $ ll /srv/downloads/zk.tar > -rw-r--r-- 1 hduser hadoop 41984000 Aug 18 13:25 /srv/downloads/zk.tar > $ hdfs dfs -put /srv/downloads/zk.tar /tmp > $ hdfs dfs -ls /tmp/zk.tar > -rw-r--r-- 2 hduser supergroup 41984000 2016-09-14 12:47 /tmp/zk.tar > I have: > -rw-r--r-- 2 hduser supergroup 41984000 2016-09-14 12:47 /tmp/zk.tar > I expect : > -rw-r--r-- 2 hduser hadoop 41984000 2016-09-14 12:47 /tmp/zk.tar > Why the HDFS group is not the value of the dfs.permissions.superusergroup > property ? > @jbigdata.fr -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9668) Optimize the locking in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HDFS-9668:
---
Attachment: HDFS-9668-12.patch
Upload a new patch V12 to fix the checkstyle warnings.
Again, the warning ( FsDatasetImpl.java:2234: @Override:3: Method length is
170 lines (max allowed is 150).) won't be fixed.
> Optimize the locking in FsDatasetImpl
> -
>
> Key: HDFS-9668
> URL: https://issues.apache.org/jira/browse/HDFS-9668
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-9668-1.patch, HDFS-9668-10.patch,
> HDFS-9668-11.patch, HDFS-9668-12.patch, HDFS-9668-2.patch, HDFS-9668-3.patch,
> HDFS-9668-4.patch, HDFS-9668-5.patch, HDFS-9668-6.patch, HDFS-9668-7.patch,
> HDFS-9668-8.patch, HDFS-9668-9.patch, execution_time.png
>
>
> During the HBase test on a tiered storage of HDFS (WAL is stored in
> SSD/RAMDISK, and all other files are stored in HDD), we observe many
> long-time BLOCKED threads on FsDatasetImpl in DataNode. The following is part
> of the jstack result:
> {noformat}
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48521 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779272_40852]" - Thread
> t@93336
>java.lang.Thread.State: BLOCKED
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:)
> - waiting to lock <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl) owned by
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" t@93335
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
>
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" - Thread
> t@93335
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:271)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1140)
> - locked <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
> {noformat}
> We measured the execution of some operations in FsDatasetImpl during the
> test. Here following is the result.
> !execution_time.png!
> The operations of finalizeBlock, addBlock and createRbw on HDD in a heavy
> load take a really long time.
> It means one slow operation of finalizeBlock, addBlock and createRbw in a
> slow storage can block all the other same operations in the same DataNode,
> especially in HBase when many wal/flusher/compactor are configured.
> We need a finer grained lock mechanism in a new FsDatasetImpl implementation
> and users can choose the implementation by configuring
> "dfs.datanode.fsdataset.factory" in DataNode.
> We can implement the lock by either storage
[jira] [Commented] (HDFS-9668) Optimize the locking in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505725#comment-15505725
]
Jingcheng Du commented on HDFS-9668:
Checked the logs of the failed unit tests, the failures should not be related
with this patch.
> Optimize the locking in FsDatasetImpl
> -
>
> Key: HDFS-9668
> URL: https://issues.apache.org/jira/browse/HDFS-9668
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-9668-1.patch, HDFS-9668-10.patch,
> HDFS-9668-11.patch, HDFS-9668-2.patch, HDFS-9668-3.patch, HDFS-9668-4.patch,
> HDFS-9668-5.patch, HDFS-9668-6.patch, HDFS-9668-7.patch, HDFS-9668-8.patch,
> HDFS-9668-9.patch, execution_time.png
>
>
> During the HBase test on a tiered storage of HDFS (WAL is stored in
> SSD/RAMDISK, and all other files are stored in HDD), we observe many
> long-time BLOCKED threads on FsDatasetImpl in DataNode. The following is part
> of the jstack result:
> {noformat}
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48521 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779272_40852]" - Thread
> t@93336
>java.lang.Thread.State: BLOCKED
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:)
> - waiting to lock <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl) owned by
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" t@93335
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
>
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" - Thread
> t@93335
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:271)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1140)
> - locked <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
> {noformat}
> We measured the execution of some operations in FsDatasetImpl during the
> test. Here following is the result.
> !execution_time.png!
> The operations of finalizeBlock, addBlock and createRbw on HDD in a heavy
> load take a really long time.
> It means one slow operation of finalizeBlock, addBlock and createRbw in a
> slow storage can block all the other same operations in the same DataNode,
> especially in HBase when many wal/flusher/compactor are configured.
> We need a finer grained lock mechanism in a new FsDatasetImpl implementation
> and users can choose the implementation by configuring
> "dfs.datanode.fsdataset.factory" in DataNode.
> We can implement the lock by either storage level or block-level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
