[jira] [Commented] (HDFS-13369) FSCK Report broken with RequestHedgingProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440394#comment-16440394
]
Ranith Sardar commented on HDFS-13369:
--
HDFS-9836 is duplicate to this issue so linking the same.
[~xyao], [~elgoiri] and [~Guocui] can you guys take look into this,if you get
chance.
> FSCK Report broken with RequestHedgingProxyProvider
>
>
> Key: HDFS-13369
> URL: https://issues.apache.org/jira/browse/HDFS-13369
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.3
>Reporter: Harshakiran Reddy
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-13369.001.patch
>
>
> Scenario:-
> 1.Configure the RequestHedgingProxy
> 2. write some files in file system
> 3. Take FSCK report for the above files
>
> {noformat}
> bin> hdfs fsck /file1 -locations -files -blocks
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider$RequestHedgingInvocationHandler
> cannot be cast to org.apache.hadoop.ipc.RpcInvocationHandler
> at org.apache.hadoop.ipc.RPC.getConnectionIdForProxy(RPC.java:626)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.getConnectionId(RetryInvocationHandler.java:438)
> at org.apache.hadoop.ipc.RPC.getConnectionIdForProxy(RPC.java:628)
> at org.apache.hadoop.ipc.RPC.getServerAddress(RPC.java:611)
> at org.apache.hadoop.hdfs.HAUtil.getAddressOfActive(HAUtil.java:263)
> at
> org.apache.hadoop.hdfs.tools.DFSck.getCurrentNamenodeAddress(DFSck.java:257)
> at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:319)
> at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:72)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:156)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:153)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:152)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:385){noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9836) RequestHedgingInvocationHandler can't be cast to org.apache.hadoop.ipc.RpcInvocationHandler
[ https://issues.apache.org/jira/browse/HDFS-9836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440390#comment-16440390 ] Ranith Sardar commented on HDFS-9836: - We've come across similar issue (HDFS-13369) and I provided patch for same, if you get chance, could please review. thanks > RequestHedgingInvocationHandler can't be cast to > org.apache.hadoop.ipc.RpcInvocationHandler > --- > > Key: HDFS-9836 > URL: https://issues.apache.org/jira/browse/HDFS-9836 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: 2.8.0 >Reporter: Guocui Mi >Assignee: Guocui Mi >Priority: Trivial > Attachments: HDFS-9836-000.patch, HDFS-9836-001.patch > > > RequestHedgingInvocationHandler cannot be cast to > org.apache.hadoop.ipc.RpcInvocationHandler > Reproduce steps: > 1: Set client failover provider as RequestHedgingProxyProvider. > > dfs.client.failover.proxy.provider.[nameservice] > > org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider > > 2: run hdfs fsck / will get following exceptions. > Exception in thread "main" java.lang.ClassCastException: > org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider$RequestHedgingInvocationHandler > cannot be cast to org.apache.hadoop.ipc.RpcInvocationHandler > at org.apache.hadoop.ipc.RPC.getConnectionIdForProxy(RPC.java:613) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.getConnectionId(RetryInvocationHandler.java:281) > at org.apache.hadoop.ipc.RPC.getConnectionIdForProxy(RPC.java:615) > at org.apache.hadoop.ipc.RPC.getServerAddress(RPC.java:598) > at org.apache.hadoop.hdfs.HAUtil.getAddressOfActive(HAUtil.java:380) > at > org.apache.hadoop.hdfs.tools.DFSck.getCurrentNamenodeAddress(DFSck.java:248) > at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:255) > at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:72) > at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:148) > at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:145) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:144) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:360) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[ https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440370#comment-16440370 ] Mohammad Arshad commented on HDFS-13443: I submitted the patch. > RBF: Update mount table cache immediately after changing (add/update/remove) > mount table entries. > - > > Key: HDFS-13443 > URL: https://issues.apache.org/jira/browse/HDFS-13443 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Mohammad Arshad >Priority: Major > Labels: RBF > Attachments: HDFS-13443-branch-2.001.patch > > > Currently mount table cache is updated periodically, by default cache is > updated every minute. After change in mount table, user operations may still > use old mount table. This is bit wrong. > To update mount table cache, maybe we can do following > * *Add refresh API in MountTableManager which will update mount table cache.* > * *When there is a change in mount table entries, router admin server can > update its cache and ask other routers to update their cache*. For example if > there are three routers R1,R2,R3 in a cluster then add mount table entry API, > at admin server side, will perform following sequence of action > ## user submit add mount table entry request on R1 > ## R1 adds the mount table entry in state store > ## R1 call refresh API on R2 > ## R1 calls refresh API on R3 > ## R1 directly freshest its cache > ## Add mount table entry response send back to user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[ https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Arshad updated HDFS-13443: --- Target Version/s: 2.10.0, 3.2.0 (was: 3.2.0) Status: Patch Available (was: Open) > RBF: Update mount table cache immediately after changing (add/update/remove) > mount table entries. > - > > Key: HDFS-13443 > URL: https://issues.apache.org/jira/browse/HDFS-13443 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Mohammad Arshad >Priority: Major > Labels: RBF > Attachments: HDFS-13443-branch-2.001.patch > > > Currently mount table cache is updated periodically, by default cache is > updated every minute. After change in mount table, user operations may still > use old mount table. This is bit wrong. > To update mount table cache, maybe we can do following > * *Add refresh API in MountTableManager which will update mount table cache.* > * *When there is a change in mount table entries, router admin server can > update its cache and ask other routers to update their cache*. For example if > there are three routers R1,R2,R3 in a cluster then add mount table entry API, > at admin server side, will perform following sequence of action > ## user submit add mount table entry request on R1 > ## R1 adds the mount table entry in state store > ## R1 call refresh API on R2 > ## R1 calls refresh API on R3 > ## R1 directly freshest its cache > ## Add mount table entry response send back to user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[ https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Arshad updated HDFS-13443: --- Attachment: HDFS-13443-branch-2.001.patch > RBF: Update mount table cache immediately after changing (add/update/remove) > mount table entries. > - > > Key: HDFS-13443 > URL: https://issues.apache.org/jira/browse/HDFS-13443 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Mohammad Arshad >Priority: Major > Labels: RBF > Attachments: HDFS-13443-branch-2.001.patch > > > Currently mount table cache is updated periodically, by default cache is > updated every minute. After change in mount table, user operations may still > use old mount table. This is bit wrong. > To update mount table cache, maybe we can do following > * *Add refresh API in MountTableManager which will update mount table cache.* > * *When there is a change in mount table entries, router admin server can > update its cache and ask other routers to update their cache*. For example if > there are three routers R1,R2,R3 in a cluster then add mount table entry API, > at admin server side, will perform following sequence of action > ## user submit add mount table entry request on R1 > ## R1 adds the mount table entry in state store > ## R1 call refresh API on R2 > ## R1 calls refresh API on R3 > ## R1 directly freshest its cache > ## Add mount table entry response send back to user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[ https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440353#comment-16440353 ] Yiqun Lin commented on HDFS-13326: -- [~ywskycn], I'm also OK with adding new option of -update. But one thing I'm concerning that this will be a incompatible change. That is say, some users used to use -add option for updating mount tables won't make sense. And they should use -update instead. We would like to add a incompatible tag for this JIRA and make this change only for branch-2 and trunk. Because there are 2.9.x, 3.0.x and 3.1.x already released, we would better not introduce this incompatible change in their subsequent minor release versions. > RBF: Improve the interfaces to modify and view mount tables > --- > > Key: HDFS-13326 > URL: https://issues.apache.org/jira/browse/HDFS-13326 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Gang Li >Priority: Minor > Attachments: HDFS-13326.000.patch, HDFS-13326.001.patch > > > From DFSRouterAdmin cmd, currently the update logic is implemented inside add > operation, where it has some limitation (e.g. cannot update "readonly" or > removing a destination). Given the RPC alreadys separate add and update > operations, it would be better to do the same in cmd level. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13435) RBF: Improve the error loggings for printing the stack trace
[
https://issues.apache.org/jira/browse/HDFS-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-13435:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.4
2.9.2
3.1.1
3.2.0
2.10.0
Status: Resolved (was: Patch Available)
Committed to trunk, branch-3.1, branch-3.0, branch-2 and branch-2.9.
Thanks [~elgoiri], [~ywskycn] and [~ajayydv] for the review and discussion!
> RBF: Improve the error loggings for printing the stack trace
>
>
> Key: HDFS-13435
> URL: https://issues.apache.org/jira/browse/HDFS-13435
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.4
>
> Attachments: HDFS-13435.001.patch, HDFS-13435.002.patch,
> HDFS-13435.003.patch
>
>
> There are many places that using {{Logger.error(String format, Object...
> arguments)}} incorrectly.
> A example:
> {code:java}
> LOG.error("Cannot remove {}", path, e);
> {code}
> The exception passed here is no meaning and won't be printed. Actually it
> should be update to
> {code:java}
> LOG.error("Cannot remove {}: {}.", path, e.getMessage());
> {code}
> or
> {code:java}
> LOG.error("Cannot remove " + path, e));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13462) Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers
[ https://issues.apache.org/jira/browse/HDFS-13462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13462: --- Status: Patch Available (was: Open) > Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers > -- > > Key: HDFS-13462 > URL: https://issues.apache.org/jira/browse/HDFS-13462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, journal-node >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HDFS-13462.000.patch > > > Allow configurable bind-host for JournalNode's HTTP and RPC servers to allow > overriding the hostname for which the server accepts connections. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13404) RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails
[
https://issues.apache.org/jira/browse/HDFS-13404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440310#comment-16440310
]
Takanobu Asanuma commented on HDFS-13404:
-
Hi, [~szetszwo]. Could you please advise me about WebHDFS?
>From HDFS-13353 and this jira, WebHDFS seems to have a little latency to
>complete some operations after executing the APIs. Is this the specification
>of WebHDFS or not desired behavior? I added
>{{fs.contract.create-visibility-delayed=true}} to the contract of WebHDFS in
>HDFS-13353. But on second thought, it might be wrong.
> RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails
> --
>
> Key: HDFS-13404
> URL: https://issues.apache.org/jira/browse/HDFS-13404
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: detailed_error.log
>
>
> This is reported by [~elgoiri].
> {noformat}
> java.io.FileNotFoundException:
> Failed to append to non-existent file /test/test/target for client 127.0.0.1
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirAppendOp.appendFile(FSDirAppendOp.java:104)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2621)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:805)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:485)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> ...
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:549)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:527)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:136)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathOutputStreamRunner$1.close(WebHdfsFileSystem.java:1013)
> at
> org.apache.hadoop.fs.contract.AbstractContractAppendTest.testRenameFileBeingAppended(AbstractContractAppendTest.java:139)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13435) RBF: Improve the error loggings for printing the stack trace
[
https://issues.apache.org/jira/browse/HDFS-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440301#comment-16440301
]
Yiqun Lin commented on HDFS-13435:
--
Thanks [~elgoiri] for the review, committing this.
> RBF: Improve the error loggings for printing the stack trace
>
>
> Key: HDFS-13435
> URL: https://issues.apache.org/jira/browse/HDFS-13435
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-13435.001.patch, HDFS-13435.002.patch,
> HDFS-13435.003.patch
>
>
> There are many places that using {{Logger.error(String format, Object...
> arguments)}} incorrectly.
> A example:
> {code:java}
> LOG.error("Cannot remove {}", path, e);
> {code}
> The exception passed here is no meaning and won't be printed. Actually it
> should be update to
> {code:java}
> LOG.error("Cannot remove {}: {}.", path, e.getMessage());
> {code}
> or
> {code:java}
> LOG.error("Cannot remove " + path, e));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13404) RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails
[
https://issues.apache.org/jira/browse/HDFS-13404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440293#comment-16440293
]
Takanobu Asanuma commented on HDFS-13404:
-
I executed the unit test many times and confirmed that it failed once in about
100. I think this is basically the same cause of HDFS-13353 that
{{WebHdfsFileSystem}} seems to have a little latency to complete some
operations even if the APIs have already been executed.
> RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails
> --
>
> Key: HDFS-13404
> URL: https://issues.apache.org/jira/browse/HDFS-13404
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: detailed_error.log
>
>
> This is reported by [~elgoiri].
> {noformat}
> java.io.FileNotFoundException:
> Failed to append to non-existent file /test/test/target for client 127.0.0.1
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirAppendOp.appendFile(FSDirAppendOp.java:104)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2621)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:805)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:485)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> ...
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:549)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:527)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:136)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathOutputStreamRunner$1.close(WebHdfsFileSystem.java:1013)
> at
> org.apache.hadoop.fs.contract.AbstractContractAppendTest.testRenameFileBeingAppended(AbstractContractAppendTest.java:139)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13388) RequestHedgingProxyProvider calls multiple configured NNs all the time
[
https://issues.apache.org/jira/browse/HDFS-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440288#comment-16440288
]
genericqa commented on HDFS-13388:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 41s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
46s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13388 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919307/HADOOP-13388.0014.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux fbf417bd2c77 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f1461b2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23960/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23960/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> RequestHedgingProxyProvider calls multiple configured NNs all the time
> --
>
>
[jira] [Commented] (HDFS-13388) RequestHedgingProxyProvider calls multiple configured NNs all the time
[ https://issues.apache.org/jira/browse/HDFS-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440234#comment-16440234 ] Jinglun commented on HDFS-13388: Hi [~elgoiri], I give an update at patch 0014. I'm comfortable committing this, please go ahead :). > RequestHedgingProxyProvider calls multiple configured NNs all the time > -- > > Key: HDFS-13388 > URL: https://issues.apache.org/jira/browse/HDFS-13388 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HADOOP-13388.0001.patch, HADOOP-13388.0002.patch, > HADOOP-13388.0003.patch, HADOOP-13388.0004.patch, HADOOP-13388.0005.patch, > HADOOP-13388.0006.patch, HADOOP-13388.0007.patch, HADOOP-13388.0008.patch, > HADOOP-13388.0009.patch, HADOOP-13388.0010.patch, HADOOP-13388.0011.patch, > HADOOP-13388.0012.patch, HADOOP-13388.0013.patch, HADOOP-13388.0014.patch > > > In HDFS-7858 RequestHedgingProxyProvider was designed to "first > simultaneously call multiple configured NNs to decide which is the active > Namenode and then for subsequent calls it will invoke the previously > successful NN ." But the current code call multiple configured NNs every time > even when we already got the successful NN. > That's because in RetryInvocationHandler.java, ProxyDescriptor's member > proxyInfo is assigned only when it is constructed or when failover occurs. > RequestHedgingProxyProvider.currentUsedProxy is null in both cases, so the > only proxy we can get is always a dynamic proxy handled by > RequestHedgingInvocationHandler.class. RequestHedgingInvocationHandler.class > handles invoked method by calling multiple configured NNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13388) RequestHedgingProxyProvider calls multiple configured NNs all the time
[ https://issues.apache.org/jira/browse/HDFS-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun updated HDFS-13388: --- Attachment: HADOOP-13388.0014.patch > RequestHedgingProxyProvider calls multiple configured NNs all the time > -- > > Key: HDFS-13388 > URL: https://issues.apache.org/jira/browse/HDFS-13388 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HADOOP-13388.0001.patch, HADOOP-13388.0002.patch, > HADOOP-13388.0003.patch, HADOOP-13388.0004.patch, HADOOP-13388.0005.patch, > HADOOP-13388.0006.patch, HADOOP-13388.0007.patch, HADOOP-13388.0008.patch, > HADOOP-13388.0009.patch, HADOOP-13388.0010.patch, HADOOP-13388.0011.patch, > HADOOP-13388.0012.patch, HADOOP-13388.0013.patch, HADOOP-13388.0014.patch > > > In HDFS-7858 RequestHedgingProxyProvider was designed to "first > simultaneously call multiple configured NNs to decide which is the active > Namenode and then for subsequent calls it will invoke the previously > successful NN ." But the current code call multiple configured NNs every time > even when we already got the successful NN. > That's because in RetryInvocationHandler.java, ProxyDescriptor's member > proxyInfo is assigned only when it is constructed or when failover occurs. > RequestHedgingProxyProvider.currentUsedProxy is null in both cases, so the > only proxy we can get is always a dynamic proxy handled by > RequestHedgingInvocationHandler.class. RequestHedgingInvocationHandler.class > handles invoked method by calling multiple configured NNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13459) Ozone: Clean-up of ozone related change from MiniDFSCluste
[
https://issues.apache.org/jira/browse/HDFS-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-13459:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
[~nandakumar131] Thanks for the contribution. I have committed this to the
feature branch
> Ozone: Clean-up of ozone related change from MiniDFSCluste
> --
>
> Key: HDFS-13459
> URL: https://issues.apache.org/jira/browse/HDFS-13459
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13459-HDFS-7240.000.patch,
> HDFS-13459-HDFS-7240.001.patch
>
>
> After HDFS-13424 {{MiniOzoneCluster}} doesn't depend on {{MiniDFSCluste}},
> ozone related changes made in MiniDFSCluste can be reverted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10429) DataStreamer interrupted warning always appears when using CLI upload file
[
https://issues.apache.org/jira/browse/HDFS-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440158#comment-16440158
]
genericqa commented on HDFS-10429:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-10429 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-10429 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832706/HDFS-10429.3.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23958/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> DataStreamer interrupted warning always appears when using CLI upload file
> ---
>
> Key: HDFS-10429
> URL: https://issues.apache.org/jira/browse/HDFS-10429
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Zhiyuan Yang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-10429.1.patch, HDFS-10429.2.patch,
> HDFS-10429.3.patch
>
>
> Every time I use 'hdfs dfs -put' upload file, this warning is printed:
> {code:java}
> 16/05/18 20:57:56 WARN hdfs.DataStreamer: Caught exception
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1245)
> at java.lang.Thread.join(Thread.java:1319)
> at
> org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:871)
> at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:519)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:696)
> {code}
> The reason is this: originally, DataStreamer::closeResponder always prints a
> warning about InterruptedException; since HDFS-9812,
> DFSOutputStream::closeImpl always forces threads to close, which causes
> InterruptedException.
> A simple fix is to use debug level log instead of warning level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13311) RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows
[
https://issues.apache.org/jira/browse/HDFS-13311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-13311:
---
Fix Version/s: 3.2.0
> RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows
> ---
>
> Key: HDFS-13311
> URL: https://issues.apache.org/jira/browse/HDFS-13311
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Labels: RBF, windows
> Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.3
>
> Attachments: HDFS-13311.000.patch
>
>
> The Windows runs show that TestRouterAdminCLI#testCreateInvalidEntry fail
> with NPE:
> {code}
> [ERROR]
> testCreateInvalidEntry(org.apache.hadoop.hdfs.server.federation.router.TestRouterAdminCLI)
> Time elapsed: 0.008 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:529)
> at
> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:568)
> at
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:174)
> at
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:156)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.hdfs.server.federation.router.TestRouterAdminCLI.testCreateInvalidEntry(TestRouterAdminCLI.java:444)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13311) RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows
[
https://issues.apache.org/jira/browse/HDFS-13311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-13311:
---
Fix Version/s: (was: 3.0.1)
(was: 3.1.0)
3.0.3
3.1.1
> RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows
> ---
>
> Key: HDFS-13311
> URL: https://issues.apache.org/jira/browse/HDFS-13311
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Labels: RBF, windows
> Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.3
>
> Attachments: HDFS-13311.000.patch
>
>
> The Windows runs show that TestRouterAdminCLI#testCreateInvalidEntry fail
> with NPE:
> {code}
> [ERROR]
> testCreateInvalidEntry(org.apache.hadoop.hdfs.server.federation.router.TestRouterAdminCLI)
> Time elapsed: 0.008 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:529)
> at
> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:568)
> at
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:174)
> at
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:156)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.hdfs.server.federation.router.TestRouterAdminCLI.testCreateInvalidEntry(TestRouterAdminCLI.java:444)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13311) RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows
[
https://issues.apache.org/jira/browse/HDFS-13311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-13311:
--
Fix Version/s: 3.0.1
2.9.2
2.10.0
3.1.0
> RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows
> ---
>
> Key: HDFS-13311
> URL: https://issues.apache.org/jira/browse/HDFS-13311
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Labels: RBF, windows
> Fix For: 3.1.0, 2.10.0, 3.0.1, 2.9.2
>
> Attachments: HDFS-13311.000.patch
>
>
> The Windows runs show that TestRouterAdminCLI#testCreateInvalidEntry fail
> with NPE:
> {code}
> [ERROR]
> testCreateInvalidEntry(org.apache.hadoop.hdfs.server.federation.router.TestRouterAdminCLI)
> Time elapsed: 0.008 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:529)
> at
> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:568)
> at
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:174)
> at
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:156)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.hdfs.server.federation.router.TestRouterAdminCLI.testCreateInvalidEntry(TestRouterAdminCLI.java:444)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13311) RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows
[
https://issues.apache.org/jira/browse/HDFS-13311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-13311:
--
Resolution: Resolved
Status: Resolved (was: Patch Available)
> RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows
> ---
>
> Key: HDFS-13311
> URL: https://issues.apache.org/jira/browse/HDFS-13311
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Labels: RBF, windows
> Attachments: HDFS-13311.000.patch
>
>
> The Windows runs show that TestRouterAdminCLI#testCreateInvalidEntry fail
> with NPE:
> {code}
> [ERROR]
> testCreateInvalidEntry(org.apache.hadoop.hdfs.server.federation.router.TestRouterAdminCLI)
> Time elapsed: 0.008 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:529)
> at
> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:568)
> at
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:174)
> at
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:156)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.hdfs.server.federation.router.TestRouterAdminCLI.testCreateInvalidEntry(TestRouterAdminCLI.java:444)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13311) RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows
[
https://issues.apache.org/jira/browse/HDFS-13311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440147#comment-16440147
]
Virajith Jalaparti commented on HDFS-13311:
---
Thanks for working on this [~elgoiri]. Committed this to branch-2, branch-2.9,
branch-3.0, branch-3.1 and trunk.
> RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows
> ---
>
> Key: HDFS-13311
> URL: https://issues.apache.org/jira/browse/HDFS-13311
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Labels: RBF, windows
> Attachments: HDFS-13311.000.patch
>
>
> The Windows runs show that TestRouterAdminCLI#testCreateInvalidEntry fail
> with NPE:
> {code}
> [ERROR]
> testCreateInvalidEntry(org.apache.hadoop.hdfs.server.federation.router.TestRouterAdminCLI)
> Time elapsed: 0.008 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:529)
> at
> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:568)
> at
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:174)
> at
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:156)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.hdfs.server.federation.router.TestRouterAdminCLI.testCreateInvalidEntry(TestRouterAdminCLI.java:444)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10429) DataStreamer interrupted warning always appears when using CLI upload file
[
https://issues.apache.org/jira/browse/HDFS-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440145#comment-16440145
]
Tim Armstrong commented on HDFS-10429:
--
[~msm] looks like this is the issue with Sqoop that you mentioned to me - seems
like some spurious logging.
> DataStreamer interrupted warning always appears when using CLI upload file
> ---
>
> Key: HDFS-10429
> URL: https://issues.apache.org/jira/browse/HDFS-10429
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Zhiyuan Yang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-10429.1.patch, HDFS-10429.2.patch,
> HDFS-10429.3.patch
>
>
> Every time I use 'hdfs dfs -put' upload file, this warning is printed:
> {code:java}
> 16/05/18 20:57:56 WARN hdfs.DataStreamer: Caught exception
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1245)
> at java.lang.Thread.join(Thread.java:1319)
> at
> org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:871)
> at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:519)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:696)
> {code}
> The reason is this: originally, DataStreamer::closeResponder always prints a
> warning about InterruptedException; since HDFS-9812,
> DFSOutputStream::closeImpl always forces threads to close, which causes
> InterruptedException.
> A simple fix is to use debug level log instead of warning level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13408) MiniDFSCluster to support being built on randomized base directory
[
https://issues.apache.org/jira/browse/HDFS-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440124#comment-16440124
]
Giovanni Matteo Fumarola commented on HDFS-13408:
-
Thanks [~surmountian] for the patch.
NIT: After creating the random DFS Base Dir you set that value to
HDFS_MINIDFS_BASEDIR. What if we have already this value set?
We should do if (conf.get(HDFS_MINIDFS_BASEDIR) == null)\{conf.set}.
> MiniDFSCluster to support being built on randomized base directory
> --
>
> Key: HDFS-13408
> URL: https://issues.apache.org/jira/browse/HDFS-13408
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Xiao Liang
>Assignee: Xiao Liang
>Priority: Major
> Labels: windows
> Attachments: HDFS-13408.000.patch
>
>
> Generated files of MiniDFSCluster during test are not properly cleaned in
> Windows, which fails all subsequent test cases using the same default
> directory (Windows does not allow other processes to delete them). By
> migrating to randomized base directories, the conflict of test path of test
> cases will be avoided, even if they are running at the same time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13311) RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows
[
https://issues.apache.org/jira/browse/HDFS-13311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440115#comment-16440115
]
Giovanni Matteo Fumarola commented on HDFS-13311:
-
Thanks [~elgoiri] for the patch. LGTM +1.
> RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows
> ---
>
> Key: HDFS-13311
> URL: https://issues.apache.org/jira/browse/HDFS-13311
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Labels: RBF, windows
> Attachments: HDFS-13311.000.patch
>
>
> The Windows runs show that TestRouterAdminCLI#testCreateInvalidEntry fail
> with NPE:
> {code}
> [ERROR]
> testCreateInvalidEntry(org.apache.hadoop.hdfs.server.federation.router.TestRouterAdminCLI)
> Time elapsed: 0.008 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:529)
> at
> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:568)
> at
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:174)
> at
> org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:156)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.hdfs.server.federation.router.TestRouterAdminCLI.testCreateInvalidEntry(TestRouterAdminCLI.java:444)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13441) DataNode missed BlockKey update from NameNode due to HeartbeatResponse was dropped
[
https://issues.apache.org/jira/browse/HDFS-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440104#comment-16440104
]
yunjiong zhao commented on HDFS-13441:
--
Unit test failure is not related to this patch.
> DataNode missed BlockKey update from NameNode due to HeartbeatResponse was
> dropped
> --
>
> Key: HDFS-13441
> URL: https://issues.apache.org/jira/browse/HDFS-13441
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 2.7.1
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Attachments: HDFS-13441.002.patch, HDFS-13441.patch
>
>
> After NameNode failover, lots of application failed due to some DataNodes
> can't re-compute password from block token.
> {code:java}
> 2018-04-11 20:10:52,448 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> hdc3-lvs01-400-1701-048.stratus.lvs.ebay.com:50010:DataXceiver error
> processing unknown operation src: /10.142.74.116:57404 dst:
> /10.142.77.45:50010
> javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
> [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password for block_token_identifier (expiryDate=1523538652448,
> keyId=1762737944, userId=hadoop,
> blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
> access modes=[WRITE]), since the required block key (keyID=1762737944)
> doesn't exist.]
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:376)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getSaslStreams(SaslDataTransferServer.java:300)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:127)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:194)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password for block_token_identifier (expiryDate=1523538652448,
> keyId=1762737944, userId=hadoop,
> blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
> access modes=[WRITE]), since the required block key (keyID=1762737944)
> doesn't exist.
> at
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrievePassword(BlockTokenSecretManager.java:382)
> at
> org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrievePassword(BlockPoolTokenSecretManager.java:79)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.buildServerPassword(SaslDataTransferServer.java:318)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$100(SaslDataTransferServer.java:73)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$2.apply(SaslDataTransferServer.java:297)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:241)
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
> ... 7 more
> {code}
>
> In the DataNode log, we didn't see DataNode update block keys around
> 2018-04-11 09:55:00 and around 2018-04-11 19:55:00.
> {code:java}
> 2018-04-10 14:51:36,424 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-10 23:55:38,420 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-11 00:51:34,792 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-11 10:51:39,403 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-11 20:51:44,422 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-12 02:54:47,855 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-12 05:55:44,456 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> {code}
> The reason is there is
[jira] [Updated] (HDFS-13462) Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers
[ https://issues.apache.org/jira/browse/HDFS-13462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HDFS-13462: -- Description: Allow configurable bind-host for JournalNode's HTTP and RPC servers to allow overriding the hostname for which the server accepts connections. > Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers > -- > > Key: HDFS-13462 > URL: https://issues.apache.org/jira/browse/HDFS-13462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, journal-node >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HDFS-13462.000.patch > > > Allow configurable bind-host for JournalNode's HTTP and RPC servers to allow > overriding the hostname for which the server accepts connections. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13462) Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers
[ https://issues.apache.org/jira/browse/HDFS-13462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HDFS-13462: -- Attachment: HDFS-13462.000.patch > Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers > -- > > Key: HDFS-13462 > URL: https://issues.apache.org/jira/browse/HDFS-13462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, journal-node >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HDFS-13462.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13462) Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers
Lukas Majercak created HDFS-13462: - Summary: Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers Key: HDFS-13462 URL: https://issues.apache.org/jira/browse/HDFS-13462 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs, journal-node Reporter: Lukas Majercak Assignee: Lukas Majercak -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13441) DataNode missed BlockKey update from NameNode due to HeartbeatResponse was dropped
[
https://issues.apache.org/jira/browse/HDFS-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440080#comment-16440080
]
genericqa commented on HDFS-13441:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 0s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 215 unchanged - 0 fixed = 216 total (was 215) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 52s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}152m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13441 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919272/HDFS-13441.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 25203e55ceaa 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2d0662c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/23957/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23957/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23957/testReport/
[jira] [Assigned] (HDFS-13355) Create IO provider for hdsl
[ https://issues.apache.org/jira/browse/HDFS-13355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar reassigned HDFS-13355: - Assignee: (was: Ajay Kumar) > Create IO provider for hdsl > --- > > Key: HDFS-13355 > URL: https://issues.apache.org/jira/browse/HDFS-13355 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: HDFS-7240 >Reporter: Ajay Kumar >Priority: Major > Fix For: HDFS-7240 > > > Create an abstraction like FileIoProvider for hdsl to handle disk failure and > other issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13439) Add test case for read block operation when it is moved
[ https://issues.apache.org/jira/browse/HDFS-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-13439: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.1 3.2.0 Status: Resolved (was: Patch Available) Committed. Thanks for the contribution [~ajayydv]. > Add test case for read block operation when it is moved > --- > > Key: HDFS-13439 > URL: https://issues.apache.org/jira/browse/HDFS-13439 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 3.2.0, 3.1.1 > > Attachments: HDFS-13439.00.patch, HDFS-13439.01.patch, > HDFS-13439.02.patch > > > Add test case for read block operation when it is moved -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13407) Ozone: Use separated version schema for Hdds/Ozone projects
[ https://issues.apache.org/jira/browse/HDFS-13407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440050#comment-16440050 ] Xiaoyu Yao commented on HDFS-13407: --- Thanks [~elek] for the update. The change looks good to me. I hit some mvn site errors when validating the patch but it turns out to be not related to this patch. +1 pending Jenkins. > Ozone: Use separated version schema for Hdds/Ozone projects > --- > > Key: HDFS-13407 > URL: https://issues.apache.org/jira/browse/HDFS-13407 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HDFS-13407-HDFS-7240.001.patch, > HDFS-13407-HDFS-7240.002.patch > > > The community is voted to manage Hdds/Ozone in-tree but with different > release cycle. To achieve this we need to separated the versions of > hdds/ozone projects from the mainline hadoop version (3.2.0-currently). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13439) Add test case for read block operation when it is moved
[
https://issues.apache.org/jira/browse/HDFS-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440045#comment-16440045
]
Arpit Agarwal commented on HDFS-13439:
--
The test failures are clearly unrelated. I'll fix the checkstyle issue while
committing.
{code}
- assertNotEquals(blkFiles[0],blkFiles2[0]);
+ assertNotEquals(blkFiles[0], blkFiles2[0]);
{code}
Will commit this shortly.
> Add test case for read block operation when it is moved
> ---
>
> Key: HDFS-13439
> URL: https://issues.apache.org/jira/browse/HDFS-13439
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-13439.00.patch, HDFS-13439.01.patch,
> HDFS-13439.02.patch
>
>
> Add test case for read block operation when it is moved
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13439) Add test case for read block operation when it is moved
[
https://issues.apache.org/jira/browse/HDFS-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440034#comment-16440034
]
genericqa commented on HDFS-13439:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 46s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 134 unchanged - 0 fixed = 135 total (was 134) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 36s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13439 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919268/HDFS-13439.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d4f89bace9d8 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2d0662c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/23956/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Created] (HDFS-13461) mvn site:site -Phdds does not work
Xiaoyu Yao created HDFS-13461:
-
Summary: mvn site:site -Phdds does not work
Key: HDFS-13461
URL: https://issues.apache.org/jira/browse/HDFS-13461
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Xiaoyu Yao
{code}[*INFO*] *Reactor Summary:*
[*INFO*]
[*INFO*] Apache Hadoop Main . *FAILURE* [
47.386 s]
...
[*ERROR*] Failed to execute goal
org.apache.maven.plugins:maven-site-plugin:3.6:site *(default-cli)* on project
hadoop-main: *failed to get report for
org.apache.maven.plugins:maven-javadoc-plugin*: Failed to execute goal on
project hadoop-hdds-server-scm: *Could not resolve dependencies for project
org.apache.hadoop:hadoop-hdds-server-scm:jar:3.2.0-SNAPSHOT: Could not find
artifact
org.apache.hadoop:hadoop-hdds-container-service:jar:tests:3.2.0-SNAPSHOT in
public (http://nexus-private.hortonworks.com/nexus/content/groups/public)* ->
[Help 1]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13422) Ozone: Fix whitespaces and license issues in HDFS-7240 branch
[ https://issues.apache.org/jira/browse/HDFS-13422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440028#comment-16440028 ] Xiaoyu Yao commented on HDFS-13422: --- +1 pending Jenkins. > Ozone: Fix whitespaces and license issues in HDFS-7240 branch > - > > Key: HDFS-13422 > URL: https://issues.apache.org/jira/browse/HDFS-13422 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-13422-HDFS-7240.001.patch, > HDFS-13422-HDFS-7240.002.patch > > > This jira will be used to fix various findbugs, javac, license and findbugs > issues in HDFS-7240 branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13460) Fix Ozone related doc links in hadoop-project/src/site/site.xml
Xiaoyu Yao created HDFS-13460: - Summary: Fix Ozone related doc links in hadoop-project/src/site/site.xml Key: HDFS-13460 URL: https://issues.apache.org/jira/browse/HDFS-13460 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Because Hdds profile is off by default, the links in the generated site will be invalid without specifying maven profile -Phdds. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13433) webhdfs requests can be routed incorrectly in federated cluster
[
https://issues.apache.org/jira/browse/HDFS-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439966#comment-16439966
]
Jitendra Nath Pandey commented on HDFS-13433:
-
+1, The patch looks good to me.
> webhdfs requests can be routed incorrectly in federated cluster
> ---
>
> Key: HDFS-13433
> URL: https://issues.apache.org/jira/browse/HDFS-13433
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Critical
> Attachments: HDFS-13433.01.patch, HDFS-13433.02.patch,
> HDFS-13433.03.patch, HDFS-13433.04.patch
>
>
> In the following HA+Federated setup with two nameservices ns1 and ns2:
> # ns1 -> namenodes nn1, nn2
> # ns2 -> namenodes nn3, nn4
> # fs.defaultFS is {{hdfs://ns1}}.
> A webhdfs request issued to nn3/nn4 will be routed to ns1. This is because
> {{setClientNamenodeAddress}} initializes {{NameNode#clientNamenodeAddress}}
> using fs.defaultFS before the config is overriden.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13444) Ozone: Fix checkstyle issues in HDFS-7240
[ https://issues.apache.org/jira/browse/HDFS-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439947#comment-16439947 ] Nanda kumar commented on HDFS-13444: Thanks [~ljain] for updating the patch. +1 LGTM. I will commit this shortly. > Ozone: Fix checkstyle issues in HDFS-7240 > - > > Key: HDFS-13444 > URL: https://issues.apache.org/jira/browse/HDFS-13444 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13444-HDFS-7240.001.patch, > HDFS-13444-HDFS-7240.002.patch, HDFS-13444-HDFS-7240.003.patch, > HDFS-7240.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13459) Ozone: Clean-up of ozone related change from MiniDFSCluste
[
https://issues.apache.org/jira/browse/HDFS-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439927#comment-16439927
]
genericqa commented on HDFS-13459:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 49s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 235 unchanged - 0 fixed = 237 total (was 235) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
25s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}176m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| |
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
|
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:b78c94f |
| JIRA Issue | HDFS-13459 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919244/HDFS-13459-HDFS-7240.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux a3c4613a141e 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-7240 / 06d228a |
| maven | version: Apache Maven 3.3.9 |
|
[jira] [Commented] (HDFS-13441) DataNode missed BlockKey update from NameNode due to HeartbeatResponse was dropped
[
https://issues.apache.org/jira/browse/HDFS-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439905#comment-16439905
]
yunjiong zhao commented on HDFS-13441:
--
Upload HDFS-13441.002.patch, which contain unit test.
> DataNode missed BlockKey update from NameNode due to HeartbeatResponse was
> dropped
> --
>
> Key: HDFS-13441
> URL: https://issues.apache.org/jira/browse/HDFS-13441
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 2.7.1
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Attachments: HDFS-13441.002.patch, HDFS-13441.patch
>
>
> After NameNode failover, lots of application failed due to some DataNodes
> can't re-compute password from block token.
> {code:java}
> 2018-04-11 20:10:52,448 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> hdc3-lvs01-400-1701-048.stratus.lvs.ebay.com:50010:DataXceiver error
> processing unknown operation src: /10.142.74.116:57404 dst:
> /10.142.77.45:50010
> javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
> [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password for block_token_identifier (expiryDate=1523538652448,
> keyId=1762737944, userId=hadoop,
> blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
> access modes=[WRITE]), since the required block key (keyID=1762737944)
> doesn't exist.]
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:376)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getSaslStreams(SaslDataTransferServer.java:300)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:127)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:194)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password for block_token_identifier (expiryDate=1523538652448,
> keyId=1762737944, userId=hadoop,
> blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
> access modes=[WRITE]), since the required block key (keyID=1762737944)
> doesn't exist.
> at
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrievePassword(BlockTokenSecretManager.java:382)
> at
> org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrievePassword(BlockPoolTokenSecretManager.java:79)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.buildServerPassword(SaslDataTransferServer.java:318)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$100(SaslDataTransferServer.java:73)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$2.apply(SaslDataTransferServer.java:297)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:241)
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
> ... 7 more
> {code}
>
> In the DataNode log, we didn't see DataNode update block keys around
> 2018-04-11 09:55:00 and around 2018-04-11 19:55:00.
> {code:java}
> 2018-04-10 14:51:36,424 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-10 23:55:38,420 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-11 00:51:34,792 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-11 10:51:39,403 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-11 20:51:44,422 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-12 02:54:47,855 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-12 05:55:44,456 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> {code}
> The reason is there is
[jira] [Commented] (HDFS-13459) Ozone: Clean-up of ozone related change from MiniDFSCluste
[
https://issues.apache.org/jira/browse/HDFS-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439903#comment-16439903
]
genericqa commented on HDFS-13459:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 235 unchanged - 0 fixed = 237 total (was 235) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
25s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}138m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestPread |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestFileCorruption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:b78c94f |
| JIRA Issue | HDFS-13459 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919250/HDFS-13459-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux ec7361f5621e 3.13.0-141-generic #190-Ubuntu SMP Fri Jan 19
12:52:38 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-7240 / 06d228a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HDFS-13441) DataNode missed BlockKey update from NameNode due to HeartbeatResponse was dropped
[
https://issues.apache.org/jira/browse/HDFS-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439897#comment-16439897
]
yunjiong zhao commented on HDFS-13441:
--
[~hexiaoqiao] , Let DataNode pull Block Key from NameNode is one choice, since
it need change protocol, if I didn't understand wrong, it will need go to
Hadoop 4.
> DataNode missed BlockKey update from NameNode due to HeartbeatResponse was
> dropped
> --
>
> Key: HDFS-13441
> URL: https://issues.apache.org/jira/browse/HDFS-13441
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 2.7.1
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Attachments: HDFS-13441.002.patch, HDFS-13441.patch
>
>
> After NameNode failover, lots of application failed due to some DataNodes
> can't re-compute password from block token.
> {code:java}
> 2018-04-11 20:10:52,448 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> hdc3-lvs01-400-1701-048.stratus.lvs.ebay.com:50010:DataXceiver error
> processing unknown operation src: /10.142.74.116:57404 dst:
> /10.142.77.45:50010
> javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
> [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password for block_token_identifier (expiryDate=1523538652448,
> keyId=1762737944, userId=hadoop,
> blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
> access modes=[WRITE]), since the required block key (keyID=1762737944)
> doesn't exist.]
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:376)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getSaslStreams(SaslDataTransferServer.java:300)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:127)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:194)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password for block_token_identifier (expiryDate=1523538652448,
> keyId=1762737944, userId=hadoop,
> blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
> access modes=[WRITE]), since the required block key (keyID=1762737944)
> doesn't exist.
> at
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrievePassword(BlockTokenSecretManager.java:382)
> at
> org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrievePassword(BlockPoolTokenSecretManager.java:79)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.buildServerPassword(SaslDataTransferServer.java:318)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$100(SaslDataTransferServer.java:73)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$2.apply(SaslDataTransferServer.java:297)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:241)
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
> ... 7 more
> {code}
>
> In the DataNode log, we didn't see DataNode update block keys around
> 2018-04-11 09:55:00 and around 2018-04-11 19:55:00.
> {code:java}
> 2018-04-10 14:51:36,424 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-10 23:55:38,420 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-11 00:51:34,792 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-11 10:51:39,403 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-11 20:51:44,422 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-12 02:54:47,855 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-12 05:55:44,456 INFO
>
[jira] [Updated] (HDFS-13441) DataNode missed BlockKey update from NameNode due to HeartbeatResponse was dropped
[
https://issues.apache.org/jira/browse/HDFS-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yunjiong zhao updated HDFS-13441:
-
Attachment: HDFS-13441.002.patch
> DataNode missed BlockKey update from NameNode due to HeartbeatResponse was
> dropped
> --
>
> Key: HDFS-13441
> URL: https://issues.apache.org/jira/browse/HDFS-13441
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 2.7.1
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Attachments: HDFS-13441.002.patch, HDFS-13441.patch
>
>
> After NameNode failover, lots of application failed due to some DataNodes
> can't re-compute password from block token.
> {code:java}
> 2018-04-11 20:10:52,448 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> hdc3-lvs01-400-1701-048.stratus.lvs.ebay.com:50010:DataXceiver error
> processing unknown operation src: /10.142.74.116:57404 dst:
> /10.142.77.45:50010
> javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
> [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password for block_token_identifier (expiryDate=1523538652448,
> keyId=1762737944, userId=hadoop,
> blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
> access modes=[WRITE]), since the required block key (keyID=1762737944)
> doesn't exist.]
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:376)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getSaslStreams(SaslDataTransferServer.java:300)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:127)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:194)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password for block_token_identifier (expiryDate=1523538652448,
> keyId=1762737944, userId=hadoop,
> blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
> access modes=[WRITE]), since the required block key (keyID=1762737944)
> doesn't exist.
> at
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrievePassword(BlockTokenSecretManager.java:382)
> at
> org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrievePassword(BlockPoolTokenSecretManager.java:79)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.buildServerPassword(SaslDataTransferServer.java:318)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$100(SaslDataTransferServer.java:73)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$2.apply(SaslDataTransferServer.java:297)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:241)
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
> ... 7 more
> {code}
>
> In the DataNode log, we didn't see DataNode update block keys around
> 2018-04-11 09:55:00 and around 2018-04-11 19:55:00.
> {code:java}
> 2018-04-10 14:51:36,424 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-10 23:55:38,420 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-11 00:51:34,792 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-11 10:51:39,403 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-11 20:51:44,422 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-12 02:54:47,855 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> 2018-04-12 05:55:44,456 INFO
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
> block keys
> {code}
> The reason is there is SocketTimeOutException when sending heartbeat to
> StandbyNameNode
>
[jira] [Commented] (HDFS-13444) Ozone: Fix checkstyle issues in HDFS-7240
[
https://issues.apache.org/jira/browse/HDFS-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439888#comment-16439888
]
genericqa commented on HDFS-13444:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 10 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} framework in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
23s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
23s{color} | {color:red} common in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} integration-test in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
55s{color} | {color:red} hadoop-hdds/common in HDFS-7240 has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} framework in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} common in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} framework in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} common in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red}
[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439869#comment-16439869 ] Arpit Agarwal commented on HDFS-13056: -- Cherry-picked to branch-3.1. Thanks for the formal binding review and commit [~xiaochen]. > Expose file-level composite CRCs in HDFS which are comparable across > different instances/layouts > > > Key: HDFS-13056 > URL: https://issues.apache.org/jira/browse/HDFS-13056 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, erasure-coding, federation, hdfs >Affects Versions: 3.0.0 >Reporter: Dennis Huo >Assignee: Dennis Huo >Priority: Major > Fix For: 3.2.0, 3.1.1 > > Attachments: HDFS-13056-branch-2.8.001.patch, > HDFS-13056-branch-2.8.002.patch, HDFS-13056-branch-2.8.003.patch, > HDFS-13056-branch-2.8.004.patch, HDFS-13056-branch-2.8.005.patch, > HDFS-13056-branch-2.8.poc1.patch, HDFS-13056.001.patch, HDFS-13056.002.patch, > HDFS-13056.003.patch, HDFS-13056.003.patch, HDFS-13056.004.patch, > HDFS-13056.005.patch, HDFS-13056.006.patch, HDFS-13056.007.patch, > HDFS-13056.008.patch, HDFS-13056.009.patch, HDFS-13056.010.patch, > HDFS-13056.011.patch, HDFS-13056.012.patch, HDFS-13056.013.patch, > HDFS-13056.014.patch, Reference_only_zhen_PPOC_hadoop2.6.X.diff, > hdfs-file-composite-crc32-v1.pdf, hdfs-file-composite-crc32-v2.pdf, > hdfs-file-composite-crc32-v3.pdf > > > FileChecksum was first introduced in > [https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then > has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are > already stored as part of datanode metadata, and the MD5 approach is used to > compute an aggregate value in a distributed manner, with individual datanodes > computing the MD5-of-CRCs per-block in parallel, and the HDFS client > computing the second-level MD5. > > A shortcoming of this approach which is often brought up is the fact that > this FileChecksum is sensitive to the internal block-size and chunk-size > configuration, and thus different HDFS files with different block/chunk > settings cannot be compared. More commonly, one might have different HDFS > clusters which use different block sizes, in which case any data migration > won't be able to use the FileChecksum for distcp's rsync functionality or for > verifying end-to-end data integrity (on top of low-level data integrity > checks applied at data transfer time). > > This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430 > during the addition of checksum support for striped erasure-coded files; > while there was some discussion of using CRC composability, it still > ultimately settled on hierarchical MD5 approach, which also adds the problem > that checksums of basic replicated files are not comparable to striped files. > > This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses > CRC composition to remain completely chunk/block agnostic, and allows > comparison between striped vs replicated files, between different HDFS > instances, and possible even between HDFS and other external storage systems. > This feature can also be added in-place to be compatible with existing block > metadata, and doesn't need to change the normal path of chunk verification, > so is minimally invasive. This also means even large preexisting HDFS > deployments could adopt this feature to retroactively sync data. A detailed > design document can be found here: > https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-13056: - Fix Version/s: 3.1.1 > Expose file-level composite CRCs in HDFS which are comparable across > different instances/layouts > > > Key: HDFS-13056 > URL: https://issues.apache.org/jira/browse/HDFS-13056 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, erasure-coding, federation, hdfs >Affects Versions: 3.0.0 >Reporter: Dennis Huo >Assignee: Dennis Huo >Priority: Major > Fix For: 3.2.0, 3.1.1 > > Attachments: HDFS-13056-branch-2.8.001.patch, > HDFS-13056-branch-2.8.002.patch, HDFS-13056-branch-2.8.003.patch, > HDFS-13056-branch-2.8.004.patch, HDFS-13056-branch-2.8.005.patch, > HDFS-13056-branch-2.8.poc1.patch, HDFS-13056.001.patch, HDFS-13056.002.patch, > HDFS-13056.003.patch, HDFS-13056.003.patch, HDFS-13056.004.patch, > HDFS-13056.005.patch, HDFS-13056.006.patch, HDFS-13056.007.patch, > HDFS-13056.008.patch, HDFS-13056.009.patch, HDFS-13056.010.patch, > HDFS-13056.011.patch, HDFS-13056.012.patch, HDFS-13056.013.patch, > HDFS-13056.014.patch, Reference_only_zhen_PPOC_hadoop2.6.X.diff, > hdfs-file-composite-crc32-v1.pdf, hdfs-file-composite-crc32-v2.pdf, > hdfs-file-composite-crc32-v3.pdf > > > FileChecksum was first introduced in > [https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then > has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are > already stored as part of datanode metadata, and the MD5 approach is used to > compute an aggregate value in a distributed manner, with individual datanodes > computing the MD5-of-CRCs per-block in parallel, and the HDFS client > computing the second-level MD5. > > A shortcoming of this approach which is often brought up is the fact that > this FileChecksum is sensitive to the internal block-size and chunk-size > configuration, and thus different HDFS files with different block/chunk > settings cannot be compared. More commonly, one might have different HDFS > clusters which use different block sizes, in which case any data migration > won't be able to use the FileChecksum for distcp's rsync functionality or for > verifying end-to-end data integrity (on top of low-level data integrity > checks applied at data transfer time). > > This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430 > during the addition of checksum support for striped erasure-coded files; > while there was some discussion of using CRC composability, it still > ultimately settled on hierarchical MD5 approach, which also adds the problem > that checksums of basic replicated files are not comparable to striped files. > > This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses > CRC composition to remain completely chunk/block agnostic, and allows > comparison between striped vs replicated files, between different HDFS > instances, and possible even between HDFS and other external storage systems. > This feature can also be added in-place to be compatible with existing block > metadata, and doesn't need to change the normal path of chunk verification, > so is minimally invasive. This also means even large preexisting HDFS > deployments could adopt this feature to retroactively sync data. A detailed > design document can be found here: > https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13439) Add test case for read block operation when it is moved
[ https://issues.apache.org/jira/browse/HDFS-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439851#comment-16439851 ] Arpit Agarwal commented on HDFS-13439: -- +1 pending Jenkins. > Add test case for read block operation when it is moved > --- > > Key: HDFS-13439 > URL: https://issues.apache.org/jira/browse/HDFS-13439 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13439.00.patch, HDFS-13439.01.patch, > HDFS-13439.02.patch > > > Add test case for read block operation when it is moved -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13439) Add test case for read block operation when it is moved
[ https://issues.apache.org/jira/browse/HDFS-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439844#comment-16439844 ] Ajay Kumar commented on HDFS-13439: --- [~arpitagarwal], thanks for review. Added suggested check in patch v2. > Add test case for read block operation when it is moved > --- > > Key: HDFS-13439 > URL: https://issues.apache.org/jira/browse/HDFS-13439 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13439.00.patch, HDFS-13439.01.patch, > HDFS-13439.02.patch > > > Add test case for read block operation when it is moved -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13439) Add test case for read block operation when it is moved
[ https://issues.apache.org/jira/browse/HDFS-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13439: -- Attachment: HDFS-13439.02.patch > Add test case for read block operation when it is moved > --- > > Key: HDFS-13439 > URL: https://issues.apache.org/jira/browse/HDFS-13439 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13439.00.patch, HDFS-13439.01.patch, > HDFS-13439.02.patch > > > Add test case for read block operation when it is moved -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[ https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439843#comment-16439843 ] Íñigo Goiri commented on HDFS-13326: I'm OK with the change itself; making a -update option makes sense. For the tests, yes, we should refactor it and try to reuse as much as possible. > RBF: Improve the interfaces to modify and view mount tables > --- > > Key: HDFS-13326 > URL: https://issues.apache.org/jira/browse/HDFS-13326 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Gang Li >Priority: Minor > Attachments: HDFS-13326.000.patch, HDFS-13326.001.patch > > > From DFSRouterAdmin cmd, currently the update logic is implemented inside add > operation, where it has some limitation (e.g. cannot update "readonly" or > removing a destination). Given the RPC alreadys separate add and update > operations, it would be better to do the same in cmd level. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13459) Ozone: Clean-up of ozone related change from MiniDFSCluste
[
https://issues.apache.org/jira/browse/HDFS-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439759#comment-16439759
]
Anu Engineer commented on HDFS-13459:
-
+1, pending Jenkins.
> Ozone: Clean-up of ozone related change from MiniDFSCluste
> --
>
> Key: HDFS-13459
> URL: https://issues.apache.org/jira/browse/HDFS-13459
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13459-HDFS-7240.000.patch,
> HDFS-13459-HDFS-7240.001.patch
>
>
> After HDFS-13424 {{MiniOzoneCluster}} doesn't depend on {{MiniDFSCluste}},
> ozone related changes made in MiniDFSCluste can be reverted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13459) Ozone: Clean-up of ozone related change from MiniDFSCluste
[
https://issues.apache.org/jira/browse/HDFS-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar updated HDFS-13459:
---
Attachment: HDFS-13459-HDFS-7240.001.patch
> Ozone: Clean-up of ozone related change from MiniDFSCluste
> --
>
> Key: HDFS-13459
> URL: https://issues.apache.org/jira/browse/HDFS-13459
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13459-HDFS-7240.000.patch,
> HDFS-13459-HDFS-7240.001.patch
>
>
> After HDFS-13424 {{MiniOzoneCluster}} doesn't depend on {{MiniDFSCluste}},
> ozone related changes made in MiniDFSCluste can be reverted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439714#comment-16439714
]
Wei Yan edited comment on HDFS-13326 at 4/16/18 4:58 PM:
-
Thanks [~gangli2384] for [^HDFS-13326.001.patch]. Please fix the checkstyle
errors, most are 80-character issues.
In *RouterAdmin.java*:
The error msg
{quote}"Cannot add destination at an existing mount point " + mount);
{quote}
can be updated to
{quote}"Cannot add destination to an existing mount point. Please use -update
cmd."
{quote}
For *TestRouterAdminCLI.java*: we can merge multiple test functions together.
Don't need to check field updates one by one :)
[~elgoiri] [~linyiqun] do u guys have any more comments? We are changing the
original "-add" cmd, just to make sure it doesn't break anything from you guy
side.
was (Author: ywskycn):
Thanks [~gangli2384] for [^HDFS-13326.001.patch]. Please fix the checkstyle
errors, most are 80-character issues.
In *RouterAdmin.java*:
The error msg
{quote}"Cannot add destination at an existing mount point " + mount);
{quote}
can be updated to
{quote}"Cannot add destination to an existing mount point. Please use -update
cmd."
{quote}
For *TestRouterAdminCLI.java*: we can merge multiple test functions together.
Don't need to check field updates one by one :)
[~elgoiri] [~linyiqun] do u guys have any more comments? We are changing the
original "-add" cmd, just to make sure it doesn't break anything from you guy
side.
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Gang Li
>Priority: Minor
> Attachments: HDFS-13326.000.patch, HDFS-13326.001.patch
>
>
> From DFSRouterAdmin cmd, currently the update logic is implemented inside add
> operation, where it has some limitation (e.g. cannot update "readonly" or
> removing a destination). Given the RPC alreadys separate add and update
> operations, it would be better to do the same in cmd level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439714#comment-16439714
]
Wei Yan commented on HDFS-13326:
Thanks [~gangli2384] for [^HDFS-13326.001.patch]. Please fix the checkstyle
errors, most are 80-character issues.
In *RouterAdmin.java*:
The error msg
{quote}"Cannot add destination at an existing mount point " + mount);
{quote}
can be updated to
{quote}"Cannot add destination to an existing mount point. Please use -update
cmd."
{quote}
For *TestRouterAdminCLI.java*: we can merge multiple test functions together.
Don't need to check field updates one by one :)
[~elgoiri] [~linyiqun] do u guys have any more comments? We are changing the
original "-add" cmd, just to make sure it doesn't break anything from you guy
side.
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Gang Li
>Priority: Minor
> Attachments: HDFS-13326.000.patch, HDFS-13326.001.patch
>
>
> From DFSRouterAdmin cmd, currently the update logic is implemented inside add
> operation, where it has some limitation (e.g. cannot update "readonly" or
> removing a destination). Given the RPC alreadys separate add and update
> operations, it would be better to do the same in cmd level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13407) Ozone: Use separated version schema for Hdds/Ozone projects
[
https://issues.apache.org/jira/browse/HDFS-13407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439707#comment-16439707
]
Elek, Marton commented on HDFS-13407:
-
Sure, I rebased it just now. Created a new build (including HADOOP-15369) and
tested with the docker based acceptance tests:
{code}
==
Acceptance
==
Acceptance.Ozone :: Smoke test to start cluster with docker-compose environ...
==
Daemons are running without error | PASS |
--
Check if datanode is connected to the scm | PASS |
--
Scale it up to 5 datanodes| PASS |
--
Test rest interface | PASS |
--
Test ozone cli| PASS |
--
Check webui static resources | PASS |
--
Start freon testing | PASS |
--
Acceptance.Ozone :: Smoke test to start cluster with docker-compos... | PASS |
7 critical tests, 7 passed, 0 failed
7 tests total, 7 passed, 0 failed
==
Acceptance| PASS |
7 critical tests, 7 passed, 0 failed
7 tests total, 7 passed, 0 failed
==
{code}
> Ozone: Use separated version schema for Hdds/Ozone projects
> ---
>
> Key: HDFS-13407
> URL: https://issues.apache.org/jira/browse/HDFS-13407
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDFS-13407-HDFS-7240.001.patch,
> HDFS-13407-HDFS-7240.002.patch
>
>
> The community is voted to manage Hdds/Ozone in-tree but with different
> release cycle. To achieve this we need to separated the versions of
> hdds/ozone projects from the mainline hadoop version (3.2.0-currently).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13407) Ozone: Use separated version schema for Hdds/Ozone projects
[ https://issues.apache.org/jira/browse/HDFS-13407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-13407: Attachment: HDFS-13407-HDFS-7240.002.patch > Ozone: Use separated version schema for Hdds/Ozone projects > --- > > Key: HDFS-13407 > URL: https://issues.apache.org/jira/browse/HDFS-13407 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HDFS-13407-HDFS-7240.001.patch, > HDFS-13407-HDFS-7240.002.patch > > > The community is voted to manage Hdds/Ozone in-tree but with different > release cycle. To achieve this we need to separated the versions of > hdds/ozone projects from the mainline hadoop version (3.2.0-currently). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13444) Ozone: Fix checkstyle issues in HDFS-7240
[ https://issues.apache.org/jira/browse/HDFS-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439704#comment-16439704 ] Lokesh Jain commented on HDFS-13444: [~nandakumar131] I have rebased and uploaded v3 patch. > Ozone: Fix checkstyle issues in HDFS-7240 > - > > Key: HDFS-13444 > URL: https://issues.apache.org/jira/browse/HDFS-13444 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13444-HDFS-7240.001.patch, > HDFS-13444-HDFS-7240.002.patch, HDFS-13444-HDFS-7240.003.patch, > HDFS-7240.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13444) Ozone: Fix checkstyle issues in HDFS-7240
[ https://issues.apache.org/jira/browse/HDFS-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13444: --- Attachment: HDFS-13444-HDFS-7240.003.patch > Ozone: Fix checkstyle issues in HDFS-7240 > - > > Key: HDFS-13444 > URL: https://issues.apache.org/jira/browse/HDFS-13444 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13444-HDFS-7240.001.patch, > HDFS-13444-HDFS-7240.002.patch, HDFS-13444-HDFS-7240.003.patch, > HDFS-7240.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13459) Ozone: Clean-up of ozone related change from MiniDFSCluste
[
https://issues.apache.org/jira/browse/HDFS-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar updated HDFS-13459:
---
Status: Patch Available (was: Open)
> Ozone: Clean-up of ozone related change from MiniDFSCluste
> --
>
> Key: HDFS-13459
> URL: https://issues.apache.org/jira/browse/HDFS-13459
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13459-HDFS-7240.000.patch
>
>
> After HDFS-13424 {{MiniOzoneCluster}} doesn't depend on {{MiniDFSCluste}},
> ozone related changes made in MiniDFSCluste can be reverted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13459) Ozone: Clean-up of ozone related change from MiniDFSCluste
[
https://issues.apache.org/jira/browse/HDFS-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar updated HDFS-13459:
---
Attachment: HDFS-13459-HDFS-7240.000.patch
> Ozone: Clean-up of ozone related change from MiniDFSCluste
> --
>
> Key: HDFS-13459
> URL: https://issues.apache.org/jira/browse/HDFS-13459
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13459-HDFS-7240.000.patch
>
>
> After HDFS-13424 {{MiniOzoneCluster}} doesn't depend on {{MiniDFSCluste}},
> ozone related changes made in MiniDFSCluste can be reverted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13459) Ozone: Clean-up of ozone related change from MiniDFSCluste
Nanda kumar created HDFS-13459:
--
Summary: Ozone: Clean-up of ozone related change from MiniDFSCluste
Key: HDFS-13459
URL: https://issues.apache.org/jira/browse/HDFS-13459
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Nanda kumar
Assignee: Nanda kumar
After HDFS-13424 {{MiniOzoneCluster}} doesn't depend on {{MiniDFSCluste}},
ozone related changes made in MiniDFSCluste can be reverted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13435) RBF: Improve the error loggings for printing the stack trace
[
https://issues.apache.org/jira/browse/HDFS-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439683#comment-16439683
]
Íñigo Goiri commented on HDFS-13435:
[^HDFS-13435.003.patch] LGTM.
We can tune these in the future if we think they are too verbose.
For now, this should be good.
> RBF: Improve the error loggings for printing the stack trace
>
>
> Key: HDFS-13435
> URL: https://issues.apache.org/jira/browse/HDFS-13435
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-13435.001.patch, HDFS-13435.002.patch,
> HDFS-13435.003.patch
>
>
> There are many places that using {{Logger.error(String format, Object...
> arguments)}} incorrectly.
> A example:
> {code:java}
> LOG.error("Cannot remove {}", path, e);
> {code}
> The exception passed here is no meaning and won't be printed. Actually it
> should be update to
> {code:java}
> LOG.error("Cannot remove {}: {}.", path, e.getMessage());
> {code}
> or
> {code:java}
> LOG.error("Cannot remove " + path, e));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13388) RequestHedgingProxyProvider calls multiple configured NNs all the time
[
https://issues.apache.org/jira/browse/HDFS-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439679#comment-16439679
]
Íñigo Goiri commented on HDFS-13388:
[^HADOOP-13388.0013.patch] LGTM.
A minor nit is that:
{code}
assertExceptionContains(
"Expected Exception Info", e);
{code}
Can fit in just one line.
Other than that this is good.
Let me know if you are comfortable committing this.
> RequestHedgingProxyProvider calls multiple configured NNs all the time
> --
>
> Key: HDFS-13388
> URL: https://issues.apache.org/jira/browse/HDFS-13388
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HADOOP-13388.0001.patch, HADOOP-13388.0002.patch,
> HADOOP-13388.0003.patch, HADOOP-13388.0004.patch, HADOOP-13388.0005.patch,
> HADOOP-13388.0006.patch, HADOOP-13388.0007.patch, HADOOP-13388.0008.patch,
> HADOOP-13388.0009.patch, HADOOP-13388.0010.patch, HADOOP-13388.0011.patch,
> HADOOP-13388.0012.patch, HADOOP-13388.0013.patch
>
>
> In HDFS-7858 RequestHedgingProxyProvider was designed to "first
> simultaneously call multiple configured NNs to decide which is the active
> Namenode and then for subsequent calls it will invoke the previously
> successful NN ." But the current code call multiple configured NNs every time
> even when we already got the successful NN.
> That's because in RetryInvocationHandler.java, ProxyDescriptor's member
> proxyInfo is assigned only when it is constructed or when failover occurs.
> RequestHedgingProxyProvider.currentUsedProxy is null in both cases, so the
> only proxy we can get is always a dynamic proxy handled by
> RequestHedgingInvocationHandler.class. RequestHedgingInvocationHandler.class
> handles invoked method by calling multiple configured NNs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13448) HDFS Block Placement - Ignore Locality for First Block Replica
[
https://issues.apache.org/jira/browse/HDFS-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439676#comment-16439676
]
Kihwal Lee commented on HDFS-13448:
---
bq. However, I think this should be a feature allowed by default. This HDFS
behavior therefore should not an extension of \{{BlockPlacementPolicyDefault}},
but a part of it.
I agree, creating yet another policy is not desirable. If anything, we should
merge existing ones.
> HDFS Block Placement - Ignore Locality for First Block Replica
> --
>
> Key: HDFS-13448
> URL: https://issues.apache.org/jira/browse/HDFS-13448
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: block placement, hdfs-client
>Affects Versions: 2.9.0, 3.0.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-13448.1.patch, HDFS-13448.2.patch
>
>
> According to the HDFS Block Place Rules:
> {quote}
> /**
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on a different node of the rack as the second replica.
> */
> {quote}
> However, there is a hint for the hdfs-client that allows the block placement
> request to not put a block replica on the local datanode _where 'local' means
> the same host as the client is being run on._
> {quote}
> /**
>* Advise that a block replica NOT be written to the local DataNode where
>* 'local' means the same host as the client is being run on.
>*
>* @see CreateFlag#NO_LOCAL_WRITE
>*/
> {quote}
> I propose that we add a new flag that allows the hdfs-client to request that
> the first block replica be placed on a random DataNode in the cluster. The
> subsequent block replicas should follow the normal block placement rules.
> The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block
> replica is not placed on the local node, but it is still placed on the local
> rack. Where this comes into play is where you have, for example, a flume
> agent that is loading data into HDFS.
> If the Flume agent is running on a DataNode, then by default, the DataNode
> local to the Flume agent will always get the first block replica and this
> leads to un-even block placements, with the local node always filling up
> faster than any other node in the cluster.
> Modifying this example, if the DataNode is removed from the host where the
> Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then
> the default block placement policy will still prefer the local rack. This
> remedies the situation only so far as now the first block replica will always
> be distributed to a DataNode on the local rack.
> This new flag would allow a single Flume agent to distribute the blocks
> randomly, evenly, over the entire cluster instead of hot-spotting the local
> node or the local rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[ https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439673#comment-16439673 ] Íñigo Goiri commented on HDFS-13443: [~arshad.mohammad], this would only be then for the local Router, right? If so, that's good; do you mind posting a patch? BTW, internally our default is 10 seconds, we may want to consider changing the default to something shorter than a minute too. > RBF: Update mount table cache immediately after changing (add/update/remove) > mount table entries. > - > > Key: HDFS-13443 > URL: https://issues.apache.org/jira/browse/HDFS-13443 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Mohammad Arshad >Priority: Major > Labels: RBF > > Currently mount table cache is updated periodically, by default cache is > updated every minute. After change in mount table, user operations may still > use old mount table. This is bit wrong. > To update mount table cache, maybe we can do following > * *Add refresh API in MountTableManager which will update mount table cache.* > * *When there is a change in mount table entries, router admin server can > update its cache and ask other routers to update their cache*. For example if > there are three routers R1,R2,R3 in a cluster then add mount table entry API, > at admin server side, will perform following sequence of action > ## user submit add mount table entry request on R1 > ## R1 adds the mount table entry in state store > ## R1 call refresh API on R2 > ## R1 calls refresh API on R3 > ## R1 directly freshest its cache > ## Add mount table entry response send back to user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13399) Make Client field AlignmentContext non-static.
[ https://issues.apache.org/jira/browse/HDFS-13399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439672#comment-16439672 ] Plamen Jeliazkov commented on HDFS-13399: - Failed unit cases appear to be unrelated. Let me know if anything stands out with the latest patch. > Make Client field AlignmentContext non-static. > -- > > Key: HDFS-13399 > URL: https://issues.apache.org/jira/browse/HDFS-13399 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-12943 >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Major > Attachments: HDFS-13399-HDFS-12943.000.patch, > HDFS-13399-HDFS-12943.001.patch, HDFS-13399-HDFS-12943.002.patch > > > In HDFS-12977, DFSClient's constructor was altered to make use of a new > static method in Client that allowed one to set an AlignmentContext. This > work is to remove that static field and make each DFSClient pass it's > AlignmentContext down to the proxy Call level. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13453) RBF: getMountPointDates should fetch latest subdir time/date when parent dir is not present but /parent/child dirs are present in mount table
[
https://issues.apache.org/jira/browse/HDFS-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439668#comment-16439668
]
Íñigo Goiri commented on HDFS-13453:
Internally, we had:
{code}
private Map getMountPointDates(String path) {
Map ret = new TreeMap<>();
if (subclusterResolver instanceof MountTableResolver) {
try {
MountTableResolver mountTable = (MountTableResolver)subclusterResolver;
List entries = mountTable.getMounts(path);
for (MountTable entry : entries) {
String srcPath = entry.getSourcePath();
long modTime = entry.getDateModified();
// Remove the suffix
srcPath = srcPath.substring(path.length());
int index = srcPath.indexOf("/");
if (index > 0) {
srcPath = srcPath.substring(0, index);
}
// Remove the / in the beginning
if (srcPath.startsWith("/")) {
srcPath = srcPath.substring(1);
}
// Get the latest date
Long prevTime = ret.get(srcPath);
if (prevTime == null || prevTime < modTime) {
ret.put(srcPath, modTime);
}
}
} catch (IOException ioe) {
LOG.error("Cannot get mount table entries", ioe);
}
}
return ret;
}
{code}
I think that [^HDFS-13453-001.patch] goes even a little further.
You may want to extract the variable and do a merge of these two approaches.
> RBF: getMountPointDates should fetch latest subdir time/date when parent dir
> is not present but /parent/child dirs are present in mount table
> -
>
> Key: HDFS-13453
> URL: https://issues.apache.org/jira/browse/HDFS-13453
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Dibyendu Karmakar
>Assignee: Dibyendu Karmakar
>Priority: Major
> Attachments: HDFS-13453-000.patch, HDFS-13453-001.patch
>
>
> [HDFS-13386|https://issues.apache.org/jira/browse/HDFS-13386] is not handling
> the case when /parent in not present in mount table but /parent/subdir is in
> mount table.
> In this case getMountPointDates is not able to fetch the latest time for
> /parent as /parent is not present in mount table.
> For this scenario we will display latest modified subdir date/time as /parent
> modified time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13431) Ozone: Ozone Shell should use RestClient and RpcClient
[
https://issues.apache.org/jira/browse/HDFS-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439638#comment-16439638
]
genericqa commented on HDFS-13431:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
53s{color} | {color:red} hadoop-hdds/common in HDFS-7240 has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
39s{color} | {color:red} hadoop-ozone/ozone-manager in HDFS-7240 has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
39s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 25m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 8s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
39s{color} | {color:red} hadoop-ozone/ozone-manager generated 19 new + 1
unchanged - 0 fixed = 20 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
5s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m 22s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} ozone-manager in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
52s{color} | {color:green} hadoop-ozone in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color}
[jira] [Commented] (HDFS-13448) HDFS Block Placement - Ignore Locality for First Block Replica
[
https://issues.apache.org/jira/browse/HDFS-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439637#comment-16439637
]
BELUGA BEHR commented on HDFS-13448:
Team, some of the unit tests that failed, were related to timeouts. I will
re-run tests to see if it's simply a matter of flaky tests. However, one was
related regarding a new configuration in the HDFS service. Also, a couple of
check-style errors were generated regarding long-lines. However, given the
current format for naming configurations, it's hard to work around and provides
little value to squeeze it in there. Few, if any, of the current
configurations adhere to this line-length restriction.
Regarding the XML configuration, I will add this configuration if you are
agreeable to the approach that I have proposed here.
{code}
java.lang.AssertionError: interface
org.apache.hadoop.hdfs.client.HdfsClientConfigKeys interface
org.apache.hadoop.hdfs.client.HdfsClientConfigKeys$StripedRead class
org.apache.hadoop.hdfs.DFSConfigKeys interface
org.apache.hadoop.hdfs.client.HdfsClientConfigKeys$BlockWrite$ReplaceDatanodeOnFailure
has 1 variables missing in hdfs-default.xml Entries:
dfs.namenode.block-placement-policy.default.allow-ignore-locality expected:<0>
but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.conf.TestConfigurationFieldsBase.testCompareConfigurationClassAgainstXml(TestConfigurationFieldsBase.java:493)
{code}
> HDFS Block Placement - Ignore Locality for First Block Replica
> --
>
> Key: HDFS-13448
> URL: https://issues.apache.org/jira/browse/HDFS-13448
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: block placement, hdfs-client
>Affects Versions: 2.9.0, 3.0.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-13448.1.patch, HDFS-13448.2.patch
>
>
> According to the HDFS Block Place Rules:
> {quote}
> /**
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on a different node of the rack as the second replica.
> */
> {quote}
> However, there is a hint for the hdfs-client that allows the block placement
> request to not put a block replica on the local datanode _where 'local' means
> the same host as the client is being run on._
> {quote}
> /**
>* Advise that a block replica NOT be written to the local DataNode where
>* 'local' means the same host as the client is being run on.
>*
>* @see CreateFlag#NO_LOCAL_WRITE
>*/
> {quote}
> I propose that we add a new flag that allows the hdfs-client to request that
> the first block replica be placed on a random DataNode in the cluster. The
> subsequent block replicas should follow the normal block placement rules.
> The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block
> replica is not placed on the local node, but it is still placed on the local
> rack. Where this comes into play is where you have, for example, a flume
> agent that is loading data into HDFS.
> If the Flume agent is running on a DataNode, then by default, the DataNode
> local to the Flume agent will always get the first block replica and this
> leads to un-even block placements, with the local node always filling up
> faster than any other node in the cluster.
> Modifying this example, if the DataNode is removed from the host where the
> Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then
> the default block placement policy will still prefer the local rack. This
> remedies the situation only so far as now the first block replica will always
> be distributed to a DataNode on the local rack.
> This new flag would allow a single Flume agent to distribute the blocks
> randomly, evenly, over the entire cluster instead of hot-spotting the local
> node or the local rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13444) Ozone: Fix checkstyle issues in HDFS-7240
[
https://issues.apache.org/jira/browse/HDFS-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439635#comment-16439635
]
genericqa commented on HDFS-13444:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
26s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
9s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} framework in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
23s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
23s{color} | {color:red} common in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
23s{color} | {color:red} integration-test in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
55s{color} | {color:red} hadoop-hdds/common in HDFS-7240 has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} framework in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} common in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} framework in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} common in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red}
[jira] [Commented] (HDFS-12615) Router-based HDFS federation phase 2
[
https://issues.apache.org/jira/browse/HDFS-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439621#comment-16439621
]
Wei Yan commented on HDFS-12615:
{quote}Looking forward to seeing the patch, :).
{quote}
Was too busy last week... Didn't have a chance to finish it... Will try to put
sth there this week..
> Router-based HDFS federation phase 2
>
>
> Key: HDFS-12615
> URL: https://issues.apache.org/jira/browse/HDFS-12615
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: RBF
>
> This umbrella JIRA tracks set of improvements over the Router-based HDFS
> federation (HDFS-10467).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13369) FSCK Report broken with RequestHedgingProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439557#comment-16439557
]
Ranith Sardar commented on HDFS-13369:
--
Thanks [~Harsha1206] for raising this issue. I have attached my patch for
solving the bug.[^HDFS-13369.001.patch]
> FSCK Report broken with RequestHedgingProxyProvider
>
>
> Key: HDFS-13369
> URL: https://issues.apache.org/jira/browse/HDFS-13369
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.3
>Reporter: Harshakiran Reddy
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-13369.001.patch
>
>
> Scenario:-
> 1.Configure the RequestHedgingProxy
> 2. write some files in file system
> 3. Take FSCK report for the above files
>
> {noformat}
> bin> hdfs fsck /file1 -locations -files -blocks
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider$RequestHedgingInvocationHandler
> cannot be cast to org.apache.hadoop.ipc.RpcInvocationHandler
> at org.apache.hadoop.ipc.RPC.getConnectionIdForProxy(RPC.java:626)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.getConnectionId(RetryInvocationHandler.java:438)
> at org.apache.hadoop.ipc.RPC.getConnectionIdForProxy(RPC.java:628)
> at org.apache.hadoop.ipc.RPC.getServerAddress(RPC.java:611)
> at org.apache.hadoop.hdfs.HAUtil.getAddressOfActive(HAUtil.java:263)
> at
> org.apache.hadoop.hdfs.tools.DFSck.getCurrentNamenodeAddress(DFSck.java:257)
> at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:319)
> at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:72)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:156)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:153)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:152)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:385){noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13369) FSCK Report broken with RequestHedgingProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ranith Sardar updated HDFS-13369:
-
Attachment: HDFS-13369.001.patch
> FSCK Report broken with RequestHedgingProxyProvider
>
>
> Key: HDFS-13369
> URL: https://issues.apache.org/jira/browse/HDFS-13369
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.3
>Reporter: Harshakiran Reddy
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-13369.001.patch
>
>
> Scenario:-
> 1.Configure the RequestHedgingProxy
> 2. write some files in file system
> 3. Take FSCK report for the above files
>
> {noformat}
> bin> hdfs fsck /file1 -locations -files -blocks
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider$RequestHedgingInvocationHandler
> cannot be cast to org.apache.hadoop.ipc.RpcInvocationHandler
> at org.apache.hadoop.ipc.RPC.getConnectionIdForProxy(RPC.java:626)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.getConnectionId(RetryInvocationHandler.java:438)
> at org.apache.hadoop.ipc.RPC.getConnectionIdForProxy(RPC.java:628)
> at org.apache.hadoop.ipc.RPC.getServerAddress(RPC.java:611)
> at org.apache.hadoop.hdfs.HAUtil.getAddressOfActive(HAUtil.java:263)
> at
> org.apache.hadoop.hdfs.tools.DFSck.getCurrentNamenodeAddress(DFSck.java:257)
> at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:319)
> at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:72)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:156)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:153)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:152)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:385){noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13424) Ozone: Refactor MiniOzoneClassicCluster
[
https://issues.apache.org/jira/browse/HDFS-13424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-13424:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
Thanks for the contribution [~nandakumar131]. I have committed this to the
feature branch.
> Ozone: Refactor MiniOzoneClassicCluster
> ---
>
> Key: HDFS-13424
> URL: https://issues.apache.org/jira/browse/HDFS-13424
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13424-HDFS-7240.000.patch,
> HDFS-13424-HDFS-7240.001.patch
>
>
> This jira will track the refactoring work on {{MiniOzoneClassicCluster}}
> which removes the dependency and changes made in {{MiniDFSCluster}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13424) Ozone: Refactor MiniOzoneClassicCluster
[
https://issues.apache.org/jira/browse/HDFS-13424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439538#comment-16439538
]
Mukul Kumar Singh commented on HDFS-13424:
--
I also ran the acceptance test and it passes successfully.
{code}
Acceptance
==
Acceptance.Ozone :: Smoke test to start cluster with docker-compose environ...
==
Daemons are running without error | PASS |
--
Check if datanode is connected to the scm | PASS |
--
Scale it up to 5 datanodes| PASS |
--
Test rest interface | PASS |
--
Test ozone cli| PASS |
--
Check webui static resources | PASS |
--
Start freon testing | PASS |
--
Acceptance.Ozone :: Smoke test to start cluster with docker-compos... | PASS |
7 critical tests, 7 passed, 0 failed
7 tests total, 7 passed, 0 failed
==
Acceptance| PASS |
7 critical tests, 7 passed, 0 failed
7 tests total, 7 passed, 0 failed
{code}
> Ozone: Refactor MiniOzoneClassicCluster
> ---
>
> Key: HDFS-13424
> URL: https://issues.apache.org/jira/browse/HDFS-13424
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13424-HDFS-7240.000.patch,
> HDFS-13424-HDFS-7240.001.patch
>
>
> This jira will track the refactoring work on {{MiniOzoneClassicCluster}}
> which removes the dependency and changes made in {{MiniDFSCluster}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13424) Ozone: Refactor MiniOzoneClassicCluster
[
https://issues.apache.org/jira/browse/HDFS-13424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439535#comment-16439535
]
Mukul Kumar Singh commented on HDFS-13424:
--
Thanks for the updated patch [~nandakumar131].
+1, the v1 patch looks good to me. I will commit this shortly.
> Ozone: Refactor MiniOzoneClassicCluster
> ---
>
> Key: HDFS-13424
> URL: https://issues.apache.org/jira/browse/HDFS-13424
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13424-HDFS-7240.000.patch,
> HDFS-13424-HDFS-7240.001.patch
>
>
> This jira will track the refactoring work on {{MiniOzoneClassicCluster}}
> which removes the dependency and changes made in {{MiniDFSCluster}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13444) Ozone: Fix checkstyle issues in HDFS-7240
[ https://issues.apache.org/jira/browse/HDFS-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439452#comment-16439452 ] Lokesh Jain commented on HDFS-13444: [~nandakumar131] Thanks for reviewing the patch! v2 patch addresses your comments. I have opened Jira HDFS-13458 for handling the javadoc issues. > Ozone: Fix checkstyle issues in HDFS-7240 > - > > Key: HDFS-13444 > URL: https://issues.apache.org/jira/browse/HDFS-13444 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13444-HDFS-7240.001.patch, > HDFS-13444-HDFS-7240.002.patch, HDFS-7240.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13458) Ozone: Fix javadoc issues in HDFS-7240
[
https://issues.apache.org/jira/browse/HDFS-13458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439447#comment-16439447
]
Lokesh Jain commented on HDFS-13458:
The issues can be listed using command
{code:java}
mvn checkstyle:checkstyle -Dcheckstyle.consoleOutput=true -Ptest-patch
-DskipTests -Ptest-patch -Phdds
{code}
> Ozone: Fix javadoc issues in HDFS-7240
> --
>
> Key: HDFS-13458
> URL: https://issues.apache.org/jira/browse/HDFS-13458
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Lokesh Jain
>Priority: Major
>
> *The following classes are missing javadoc*
> * HddsConfServlet#BadFormatException
> * HddsConfigKeys
> * BenchMarkMetadataStoreWrites
> * BenchMarkRocksDbStore
> * BenchMarkDatanodeDispatcher
> * BenchMarkContainerStateMap
> * BenchMarkMetadataStoreReads
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13458) Ozone: Fix javadoc issues in HDFS-7240
[ https://issues.apache.org/jira/browse/HDFS-13458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13458: --- Description: *The following classes are missing javadoc* * HddsConfServlet#BadFormatException * HddsConfigKeys * BenchMarkMetadataStoreWrites * BenchMarkRocksDbStore * BenchMarkDatanodeDispatcher * BenchMarkContainerStateMap * BenchMarkMetadataStoreReads > Ozone: Fix javadoc issues in HDFS-7240 > -- > > Key: HDFS-13458 > URL: https://issues.apache.org/jira/browse/HDFS-13458 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Lokesh Jain >Priority: Major > > *The following classes are missing javadoc* > * HddsConfServlet#BadFormatException > * HddsConfigKeys > * BenchMarkMetadataStoreWrites > * BenchMarkRocksDbStore > * BenchMarkDatanodeDispatcher > * BenchMarkContainerStateMap > * BenchMarkMetadataStoreReads -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13458) Ozone: Fix javadoc issues in HDFS-7240
Lokesh Jain created HDFS-13458: -- Summary: Ozone: Fix javadoc issues in HDFS-7240 Key: HDFS-13458 URL: https://issues.apache.org/jira/browse/HDFS-13458 Project: Hadoop HDFS Issue Type: Bug Reporter: Lokesh Jain -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13444) Ozone: Fix checkstyle issues in HDFS-7240
[ https://issues.apache.org/jira/browse/HDFS-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13444: --- Attachment: HDFS-13444-HDFS-7240.002.patch > Ozone: Fix checkstyle issues in HDFS-7240 > - > > Key: HDFS-13444 > URL: https://issues.apache.org/jira/browse/HDFS-13444 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13444-HDFS-7240.001.patch, > HDFS-13444-HDFS-7240.002.patch, HDFS-7240.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13431) Ozone: Ozone Shell should use RestClient and RpcClient
[ https://issues.apache.org/jira/browse/HDFS-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439431#comment-16439431 ] Lokesh Jain commented on HDFS-13431: [~msingh] I have rebased and uploaded v2 patch. > Ozone: Ozone Shell should use RestClient and RpcClient > -- > > Key: HDFS-13431 > URL: https://issues.apache.org/jira/browse/HDFS-13431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13431-HDFS-7240.001.patch, > HDFS-13431-HDFS-7240.002.patch > > > Currently Ozone Shell uses OzoneRestClient. We should use both RestClient and > RpcClient instead of OzoneRestClient. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13431) Ozone: Ozone Shell should use RestClient and RpcClient
[ https://issues.apache.org/jira/browse/HDFS-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13431: --- Attachment: HDFS-13431-HDFS-7240.002.patch > Ozone: Ozone Shell should use RestClient and RpcClient > -- > > Key: HDFS-13431 > URL: https://issues.apache.org/jira/browse/HDFS-13431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13431-HDFS-7240.001.patch, > HDFS-13431-HDFS-7240.002.patch > > > Currently Ozone Shell uses OzoneRestClient. We should use both RestClient and > RpcClient instead of OzoneRestClient. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13424) Ozone: Refactor MiniOzoneClassicCluster
[
https://issues.apache.org/jira/browse/HDFS-13424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439415#comment-16439415
]
genericqa commented on HDFS-13424:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 49 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
27s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
32s{color} | {color:red} integration-test in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
24s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
25s{color} | {color:red} hadoop-ozone in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
24s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
24s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
25s{color} | {color:red} hadoop-ozone in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
25s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
24s{color} | {color:red} integration-test in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
25s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
24s{color} | {color:red} hadoop-ozone in HDFS-7240 failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
11s{color} | {color:red} container-service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
11s{color} | {color:red} integration-test in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
10s{color} | {color:red} tools in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
11s{color} | {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
14s{color} | {color:red} container-service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
17s{color} | {color:red} integration-test in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
15s{color} | {color:red} tools in the patch failed. {color} |
|
[jira] [Created] (HDFS-13457) LocalFilesystem#rename(Path, Path, Options.Rename...) does not handle crc files
Andras Bokor created HDFS-13457: --- Summary: LocalFilesystem#rename(Path, Path, Options.Rename...) does not handle crc files Key: HDFS-13457 URL: https://issues.apache.org/jira/browse/HDFS-13457 Project: Hadoop HDFS Issue Type: Bug Reporter: Andras Bokor Assignee: Andras Bokor ChecksumFilesystem#rename(Path, Path, Options.Rename...) is missing and FilterFileSystem does not care with crc files. That causes abandoned crc files in case of rename. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13444) Ozone: Fix checkstyle issues in HDFS-7240
[
https://issues.apache.org/jira/browse/HDFS-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439372#comment-16439372
]
Nanda kumar commented on HDFS-13444:
Thanks [~ljain] for working on this. The patch looks pretty good to me, some
minor comments.
*The following classes are missing javadoc*
* HddsConfServlet#BadFormatException
* HddsConfigKeys
* BenchMarkMetadataStoreWrites
* BenchMarkRocksDbStore
* BenchMarkDatanodeDispatcher
* BenchMarkContainerStateMap
* BenchMarkMetadataStoreReads
RegisteredCommand.java:198 'hostname' hides a field - rename the setter method
argument to 'host'
RegisteredCommand.java:203 'ipAddress' hides a field - rename the setter method
argument to 'ip'
*Lines length more than 80 characters*
ozShell/Shell.java:201
ozone/MiniOzoneCluster.java:49
*BenchMarkContainerStateMap.java*
Make both {{stateMap}} and {{containerID}} as private.
*TestContainerStateManager.java:116*
Variable name {{stateManager}} is already used at the class lever, rename it to
{{containerStateManager}}
*Unused imports*
BenchMarkContainerStateMap.java:48 & 49
> Ozone: Fix checkstyle issues in HDFS-7240
> -
>
> Key: HDFS-13444
> URL: https://issues.apache.org/jira/browse/HDFS-13444
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>Priority: Major
> Attachments: HDFS-13444-HDFS-7240.001.patch, HDFS-7240.007.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13444) Ozone: Fix checkstyle issues in HDFS-7240
[
https://issues.apache.org/jira/browse/HDFS-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439369#comment-16439369
]
genericqa commented on HDFS-13444:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
7s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
25s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} framework in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
25s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
23s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
25s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} common in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
24s{color} | {color:red} integration-test in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
23s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
56s{color} | {color:red} hadoop-hdds/common in HDFS-7240 has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} framework in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} common in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} framework in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} common in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red}
[jira] [Updated] (HDFS-13449) Ozone: TestOzoneConfigurationFields is failing
[
https://issues.apache.org/jira/browse/HDFS-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiXin Ge updated HDFS-13449:
Attachment: (was: HDFS-13449-HDFS-7240.001.patch)
> Ozone: TestOzoneConfigurationFields is failing
> --
>
> Key: HDFS-13449
> URL: https://issues.apache.org/jira/browse/HDFS-13449
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: LiXin Ge
>Priority: Major
> Attachments: HDFS-13449-HDFS-7240.000.patch
>
>
> {{TestOzoneConfigurationFields}} is failing because of two properties
> introduced in ozone-default.xml by HDFS-13197
> * hadoop.custom.tags
> * ozone.system.tags
>
> Which are not present in any ConfigurationClasses.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13449) Ozone: TestOzoneConfigurationFields is failing
[
https://issues.apache.org/jira/browse/HDFS-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439353#comment-16439353
]
LiXin Ge commented on HDFS-13449:
-
The checkstyle rules here is odd...
I think the patch 000 already has the correct indent, patch 001 is deleted now.
> Ozone: TestOzoneConfigurationFields is failing
> --
>
> Key: HDFS-13449
> URL: https://issues.apache.org/jira/browse/HDFS-13449
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: LiXin Ge
>Priority: Major
> Attachments: HDFS-13449-HDFS-7240.000.patch
>
>
> {{TestOzoneConfigurationFields}} is failing because of two properties
> introduced in ozone-default.xml by HDFS-13197
> * hadoop.custom.tags
> * ozone.system.tags
>
> Which are not present in any ConfigurationClasses.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13449) Ozone: TestOzoneConfigurationFields is failing
[
https://issues.apache.org/jira/browse/HDFS-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439332#comment-16439332
]
genericqa commented on HDFS-13449:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m
26s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
59s{color} | {color:red} hadoop-hdds/common in HDFS-7240 has 1 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-hdds/common: The patch generated 2 new +
20 unchanged - 1 fixed = 22 total (was 21) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
9s{color} | {color:green} common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
25s{color} | {color:red} The patch generated 44 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:b78c94f |
| JIRA Issue | HDFS-13449 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919199/HDFS-13449-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 6209d3f1f935 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-7240 / fd84dea |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/23948/artifact/out/branch-findbugs-hadoop-hdds_common-warnings.html
|
| checkstyle |
[jira] [Updated] (HDFS-13456) Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-4309324-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HDFS-13456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13456: --- Issue Type: Improvement (was: Bug) > Ozone: Update ozone to latest ratis snapshot build > (0.1.1-alpha-4309324-SNAPSHOT) > - > > Key: HDFS-13456 > URL: https://issues.apache.org/jira/browse/HDFS-13456 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13456-HDFS-7240.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13333) Ozone: Introduce a new SCM Exception which will be thrown when mandatory property is missing
[ https://issues.apache.org/jira/browse/HDFS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439309#comment-16439309 ] LiXin Ge commented on HDFS-1: - [~nandakumar131] Thanks for paying attention on this work. I'm back you up on your idea of introducing a new interface. > Ozone: Introduce a new SCM Exception which will be thrown when mandatory > property is missing > - > > Key: HDFS-1 > URL: https://issues.apache.org/jira/browse/HDFS-1 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nanda kumar >Assignee: LiXin Ge >Priority: Major > Labels: newbie > Attachments: HDFS-1-HDFS-7240.001.patch, > HDFS-1-HDFS-7240.002.patch, HDFS-1-HDFS-7240.003.patch, > HDFS-1.001.patch > > > It's better to have a separate SCM Exception to indicate a missing mandatory > property. This was proposed by [~xyao] in [this comment| > https://issues.apache.org/jira/browse/HDFS-13300?focusedCommentId=16408553=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16408553] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13456) Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-4309324-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HDFS-13456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13456: --- Attachment: HDFS-13456-HDFS-7240.001.patch > Ozone: Update ozone to latest ratis snapshot build > (0.1.1-alpha-4309324-SNAPSHOT) > - > > Key: HDFS-13456 > URL: https://issues.apache.org/jira/browse/HDFS-13456 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13456-HDFS-7240.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13456) Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-4309324-SNAPSHOT)
Lokesh Jain created HDFS-13456: -- Summary: Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-4309324-SNAPSHOT) Key: HDFS-13456 URL: https://issues.apache.org/jira/browse/HDFS-13456 Project: Hadoop HDFS Issue Type: Bug Reporter: Lokesh Jain Assignee: Lokesh Jain -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13424) Ozone: Refactor MiniOzoneClassicCluster
[
https://issues.apache.org/jira/browse/HDFS-13424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439291#comment-16439291
]
Nanda kumar commented on HDFS-13424:
Updated patch v001 to address nitpicks and created HDFS-13455 to add test-cases
which will cover the scenarios that you have mentioned.
> Ozone: Refactor MiniOzoneClassicCluster
> ---
>
> Key: HDFS-13424
> URL: https://issues.apache.org/jira/browse/HDFS-13424
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13424-HDFS-7240.000.patch,
> HDFS-13424-HDFS-7240.001.patch
>
>
> This jira will track the refactoring work on {{MiniOzoneClassicCluster}}
> which removes the dependency and changes made in {{MiniDFSCluster}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13424) Ozone: Refactor MiniOzoneClassicCluster
[
https://issues.apache.org/jira/browse/HDFS-13424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar updated HDFS-13424:
---
Attachment: HDFS-13424-HDFS-7240.001.patch
> Ozone: Refactor MiniOzoneClassicCluster
> ---
>
> Key: HDFS-13424
> URL: https://issues.apache.org/jira/browse/HDFS-13424
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13424-HDFS-7240.000.patch,
> HDFS-13424-HDFS-7240.001.patch
>
>
> This jira will track the refactoring work on {{MiniOzoneClassicCluster}}
> which removes the dependency and changes made in {{MiniDFSCluster}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13455) Ozone: Add unit test to verify loading of HddsDatanodeService as part of datanode startup when ozone is enabled
Nanda kumar created HDFS-13455:
--
Summary: Ozone: Add unit test to verify loading of
HddsDatanodeService as part of datanode startup when ozone is enabled
Key: HDFS-13455
URL: https://issues.apache.org/jira/browse/HDFS-13455
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Nanda kumar
Assignee: Nanda kumar
When {{ozone}} is enabled and {{HddsDatanodeService}} plugin is configured,
datanode should load {{HddsDatanodeService}} plugin and start Hdds services. We
have to add unit test case inside {{hadoop/hadoop-ozone/integration-test}}
using {{MiniDFSCluster}} to verify the following scenarios
* HddsDatanodeService is loaded as part of datanode startup.
* run Freon test against {{MiniDFSCluster}} and validate the data that has
been written.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13454) Ozone : Fix the test logic in TestKeySpaceManager#testDeleteKey
Shashikant Banerjee created HDFS-13454:
--
Summary: Ozone : Fix the test logic in
TestKeySpaceManager#testDeleteKey
Key: HDFS-13454
URL: https://issues.apache.org/jira/browse/HDFS-13454
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee
Fix For: HDFS-7240
The test logic in TestKeySpaceManager#testDeleteKey seems to be wrong. The test
validates the keyArgs instead of blockId to make sure the key gets deleted from
SCM. Also, after the first exception validation , the subsequent statements in
the junit never gets executed here.
{code:java}
keys.add(keyArgs.getResourceName());
exception.expect(IOException.class);
exception.expectMessage("Specified block key does not exist");
cluster.getStorageContainerManager().getBlockLocations(keys);
// Delete the key again to test deleting non-existing key.
// These will never get executed.
exception.expect(IOException.class);
exception.expectMessage("KEY_NOT_FOUND");
storageHandler.deleteKey(keyArgs);
Assert.assertEquals(1 + numKeyDeleteFails,
ksmMetrics.getNumKeyDeletesFails());{code}
The test needs to be modified to address all these.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13279) Datanodes usage is imbalanced if number of nodes per rack is not equal
[
https://issues.apache.org/jira/browse/HDFS-13279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439275#comment-16439275
]
genericqa commented on HDFS-13279:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 31m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 30m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 30m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 4s{color} | {color:orange} root: The patch generated 9 new + 39 unchanged -
1 fixed = 48 total (was 40) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
42s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}111m 44s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}264m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.blockmanagement.TestWeightedRackBlockPlacementPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13279 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919168/HDFS-13279.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux f8a6fe236faf
