[jira] [Commented] (HDFS-9323) Randomize the DFSStripedOutputStreamWithFailure tests
[ https://issues.apache.org/jira/browse/HDFS-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980324#comment-14980324 ] Tsz Wo Nicholas Sze commented on HDFS-9323: --- > But not all block index(0 to 8) is boundary value. Can we randomly choose > some of them to run as well? Do you mean the dnIndex in runTest or something else? > Randomize the DFSStripedOutputStreamWithFailure tests > - > > Key: HDFS-9323 > URL: https://issues.apache.org/jira/browse/HDFS-9323 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Attachments: h9323_20151028.patch > > > Currently, the DFSStripedOutputStreamWithFailure tests are run with fixed > indices #0 - #19 and #59, and also a test with random index > (testDatanodeFailureRandomLength). We should randomize all the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9336) deleteSnapshot throws NPE when snapshotname is null
Brahma Reddy Battula created HDFS-9336:
--
Summary: deleteSnapshot throws NPE when snapshotname is null
Key: HDFS-9336
URL: https://issues.apache.org/jira/browse/HDFS-9336
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$DeleteSnapshotRequestProto$Builder.setSnapshotName(ClientNamenodeProtocolProtos.java:17509)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.deleteSnapshot(ClientNamenodeProtocolTranslatorPB.java:1005)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:255)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
at com.sun.proxy.$Proxy15.deleteSnapshot(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.deleteSnapshot(DFSClient.java:2106)
at
org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1660)
at
org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.deleteSnapshot(DistributedFileSystem.java:1677)
at
org.apache.hadoop.hdfs.web.TestWebHDFS.testWebHdfsAllowandDisallowSnapshots(TestWebHDFS.java:380)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8498) Blocks can be committed with wrong size
[ https://issues.apache.org/jira/browse/HDFS-8498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980481#comment-14980481 ] Daryn Sharp commented on HDFS-8498: --- I don't have any more details than in the description. We've seen the bug only once (that we know of). I looked at the code and described the bug I saw in it. > Blocks can be committed with wrong size > --- > > Key: HDFS-8498 > URL: https://issues.apache.org/jira/browse/HDFS-8498 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.5.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > > When an IBR for a UC block arrives, the NN updates the expected location's > block and replica state _only_ if it's on an unexpected storage for an > expected DN. If it's for an expected storage, only the genstamp is updated. > When the block is committed, and the expected locations are verified, only > the genstamp is checked. The size is not checked but it wasn't updated in > the expected locations anyway. > A faulty client may misreport the size when committing the block. The block > is effectively corrupted. If the NN issues replications, the received IBR is > considered corrupt, the NN invalidates the block, immediately issues another > replication. The NN eventually realizes all the original replicas are > corrupt after full BRs are received from the original DNs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9057) allow/disallow snapshots via webhdfs
[ https://issues.apache.org/jira/browse/HDFS-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980373#comment-14980373 ] Vinayakumar B commented on HDFS-9057: - This is little strange. javac passes in both JDKs, But mvn install fails with compilation error. > allow/disallow snapshots via webhdfs > > > Key: HDFS-9057 > URL: https://issues.apache.org/jira/browse/HDFS-9057 > Project: Hadoop HDFS > Issue Type: New Feature > Components: webhdfs >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Brahma Reddy Battula > Attachments: HDFS-9057-002.patch, HDFS-9057-003.patch, > HDFS-9057-004.patch, HDFS-9057.patch > > > We should be able to allow and disallow directories for snapshotting via > WebHDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9336) deleteSnapshot throws NPE when snapshotname is null
[
https://issues.apache.org/jira/browse/HDFS-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980430#comment-14980430
]
Brahma Reddy Battula commented on HDFS-9336:
Uploaded patch.. Kindly review...
> deleteSnapshot throws NPE when snapshotname is null
> ---
>
> Key: HDFS-9336
> URL: https://issues.apache.org/jira/browse/HDFS-9336
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9336.patch
>
>
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$DeleteSnapshotRequestProto$Builder.setSnapshotName(ClientNamenodeProtocolProtos.java:17509)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.deleteSnapshot(ClientNamenodeProtocolTranslatorPB.java:1005)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:255)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy15.deleteSnapshot(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.deleteSnapshot(DFSClient.java:2106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1660)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.deleteSnapshot(DistributedFileSystem.java:1677)
> at
> org.apache.hadoop.hdfs.web.TestWebHDFS.testWebHdfsAllowandDisallowSnapshots(TestWebHDFS.java:380)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liangliang Gu updated HDFS-9276:
Attachment: HDFS-9276.07.patch
> Failed to Update HDFS Delegation Token for long running application in HA mode
> --
>
> Key: HDFS-9276
> URL: https://issues.apache.org/jira/browse/HDFS-9276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, ha, security
>Affects Versions: 2.7.1
>Reporter: Liangliang Gu
>Assignee: Liangliang Gu
> Attachments: HDFS-9276.01.patch, HDFS-9276.02.patch,
> HDFS-9276.03.patch, HDFS-9276.04.patch, HDFS-9276.05.patch,
> HDFS-9276.06.patch, HDFS-9276.07.patch, debug1.PNG, debug2.PNG
>
>
> The Scenario is as follows:
> 1. NameNode HA is enabled.
> 2. Kerberos is enabled.
> 3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with
> NameNode.
> 4. We want to update the HDFS Delegation Token for long running applicatons.
> HDFS Client will generate private tokens for each NameNode. When we update
> the HDFS Delegation Token, these private tokens will not be updated, which
> will cause token expired.
> This bug can be reproduced by the following program:
> {code}
> import java.security.PrivilegedExceptionAction
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.security.UserGroupInformation
> object HadoopKerberosTest {
> def main(args: Array[String]): Unit = {
> val keytab = "/path/to/keytab/xxx.keytab"
> val principal = "x...@abc.com"
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> val ugi = UserGroupInformation.createRemoteUser("test")
> ugi.addCredentials(creds1)
> ugi.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> var i = 0
> while (true) {
> val creds1 = new org.apache.hadoop.security.Credentials()
> val ugi1 =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
> ugi1.doAs(new PrivilegedExceptionAction[Void] {
> // Get a copy of the credentials
> override def run(): Void = {
> val fs = FileSystem.get(new Configuration())
> fs.addDelegationTokens("test", creds1)
> null
> }
> })
> UserGroupInformation.getCurrentUser.addCredentials(creds1)
> val fs = FileSystem.get( new Configuration())
> i += 1
> println()
> println(i)
> println(fs.listFiles(new Path("/user"), false))
> Thread.sleep(60 * 1000)
> }
> null
> }
> })
> }
> }
> {code}
> To reproduce the bug, please set the following configuration to Name Node:
> {code}
> dfs.namenode.delegation.token.max-lifetime = 10min
> dfs.namenode.delegation.key.update-interval = 3min
> dfs.namenode.delegation.token.renew-interval = 3min
> {code}
> The bug will occure after 3 minutes.
> The stacktrace is:
> {code}
> Exception in thread "main"
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1679)
> at
>
[jira] [Updated] (HDFS-9336) deleteSnapshot throws NPE when snapshotname is null
[
https://issues.apache.org/jira/browse/HDFS-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-9336:
---
Status: Patch Available (was: Open)
> deleteSnapshot throws NPE when snapshotname is null
> ---
>
> Key: HDFS-9336
> URL: https://issues.apache.org/jira/browse/HDFS-9336
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9336.patch
>
>
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$DeleteSnapshotRequestProto$Builder.setSnapshotName(ClientNamenodeProtocolProtos.java:17509)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.deleteSnapshot(ClientNamenodeProtocolTranslatorPB.java:1005)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:255)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy15.deleteSnapshot(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.deleteSnapshot(DFSClient.java:2106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1660)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.deleteSnapshot(DistributedFileSystem.java:1677)
> at
> org.apache.hadoop.hdfs.web.TestWebHDFS.testWebHdfsAllowandDisallowSnapshots(TestWebHDFS.java:380)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9336) deleteSnapshot throws NPE when snapshotname is null
[
https://issues.apache.org/jira/browse/HDFS-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-9336:
---
Attachment: HDFS-9336.patch
> deleteSnapshot throws NPE when snapshotname is null
> ---
>
> Key: HDFS-9336
> URL: https://issues.apache.org/jira/browse/HDFS-9336
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9336.patch
>
>
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$DeleteSnapshotRequestProto$Builder.setSnapshotName(ClientNamenodeProtocolProtos.java:17509)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.deleteSnapshot(ClientNamenodeProtocolTranslatorPB.java:1005)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:255)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy15.deleteSnapshot(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.deleteSnapshot(DFSClient.java:2106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1660)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.deleteSnapshot(DistributedFileSystem.java:1677)
> at
> org.apache.hadoop.hdfs.web.TestWebHDFS.testWebHdfsAllowandDisallowSnapshots(TestWebHDFS.java:380)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9057) allow/disallow snapshots via webhdfs
[
https://issues.apache.org/jira/browse/HDFS-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980311#comment-14980311
]
Hadoop QA commented on HDFS-9057:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 59s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 39s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 28s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 23s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hadoop-hdfs-project (total was 183, now 186). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 36s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 40s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 20s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 147m 3s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_79 Failed junit tests | hadoop.hdfs.server.namenode.TestDeleteRace
|
| | hadoop.hdfs.server.blockmanagement.TestNodeCount |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| |
[jira] [Commented] (HDFS-9261) Erasure Coding: Skip encoding the data cells if all the parity data streamers are failed for the current block group
[
https://issues.apache.org/jira/browse/HDFS-9261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980436#comment-14980436
]
Hudson commented on HDFS-9261:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #550 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/550/])
HDFS-9261. Erasure Coding: Skip encoding the data cells if all the (umamahesh:
rev 07ecdb877dee0aa76a3269e37ac4783a55ab6bf6)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
> Erasure Coding: Skip encoding the data cells if all the parity data streamers
> are failed for the current block group
>
>
> Key: HDFS-9261
> URL: https://issues.apache.org/jira/browse/HDFS-9261
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HDFS-9261-00.patch, HDFS-9261-01.patch,
> HDFS-9261-02.patch
>
>
> {{DFSStripedOutputStream}} will continue writing with minimum number
> (dataBlockNum) of live datanodes. It won't replace the failed datanodes
> immediately for the current block group. Consider a case where all the parity
> data streamers are failed, now it is unnecessary to encode the data block
> cells and generate the parity data. This is a corner case where it can skip
> {{writeParityCells()}} step.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8545) Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total file length from a specific path
[ https://issues.apache.org/jira/browse/HDFS-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980437#comment-14980437 ] Hudson commented on HDFS-8545: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #550 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/550/]) HDFS-8545. Refactor FS#getUsed() to use ContentSummary and add an API to (vinayakumarb: rev 7d2d16f4ee87ae56dc20016a91c109dd5130f7d4) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java > Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total > file length from a specific path > -- > > Key: HDFS-8545 > URL: https://issues.apache.org/jira/browse/HDFS-8545 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: J.Andreina >Assignee: J.Andreina >Priority: Minor > Fix For: 2.8.0 > > Attachments: HDFS-8545.1.patch, HDFS-8545.2.patch, HDFS-8545.3.patch, > HDFS-8545.4.patch, HDFS-8545.5.patch, HDFS-8545.6.patch, HDFS-8545.7.patch > > > Currently by default in FileSystem#getUsed() returns the total file size from > root. > It is good to have an api to return the total file size from specified path > ,same as we specify the path in "./hdfs dfs -du -s /path" . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980438#comment-14980438
]
Hudson commented on HDFS-9044:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #550 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/550/])
HDFS-9044. Give Priority to FavouredNodes , before selecting nodes from
(vinayakumarb: rev 588baab160e7c328dca1c45cf3541e79218406e8)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Fix For: 2.8.0
>
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch,
> HDFS-9044.4.patch, HDFS-9044.5.patch, HDFS-9044.6.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9339) Extend full test of KMS ACLs
[ https://issues.apache.org/jira/browse/HDFS-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-9339: --- Status: Patch Available (was: Open) [~zhz], can you take a look when you get a chance? > Extend full test of KMS ACLs > > > Key: HDFS-9339 > URL: https://issues.apache.org/jira/browse/HDFS-9339 > Project: Hadoop HDFS > Issue Type: Test > Components: HDFS >Affects Versions: 2.7.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: HDFS-9339.001.patch > > > HDFS-9295 adds an end-to-end test for KMS, but it is missing a dimension. > The tests added in that JIRA hold the configuration constant and test that > all operations succeed or fail as expected. More tests are needed that hold > the operation constant and test that all possible configurations cause the > operations to succeed or fail as expected. This JIRA is to add those tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9336) deleteSnapshot throws NPE when snapshotname is null
[
https://issues.apache.org/jira/browse/HDFS-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980700#comment-14980700
]
Hadoop QA commented on HDFS-9336:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 6s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
5s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 2s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s
{color} | {color:red} Patch generated 1 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 57, now 57). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 55s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 25s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 19s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 154m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_79 Failed junit tests |
hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.0 Server=1.7.0
Image:test-patch-base-hadoop-date2015-10-29 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12769536/HDFS-9336.patch |
| JIRA Issue | HDFS-9336 |
| Optional Tests | asflicense javac javadoc mvninstall unit findbugs
checkstyle compile |
| uname | Linux 87896e61df92 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-67f42f1/precommit/personality/hadoop.sh
|
| git revision | trunk / c416999 |
[jira] [Commented] (HDFS-9336) deleteSnapshot throws NPE when snapshotname is null
[
https://issues.apache.org/jira/browse/HDFS-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980609#comment-14980609
]
Rakesh R commented on HDFS-9336:
Thanks for the work [~brahmareddy]. I've a quick comment.
In tests, please use {{GenericTestUtils.assertExceptionContains}} instead of
directly using e.getMessage.contains(""). Could you refer
[TestDataStorage.java#L173|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataStorage.java#L173]
to know the usage.
> deleteSnapshot throws NPE when snapshotname is null
> ---
>
> Key: HDFS-9336
> URL: https://issues.apache.org/jira/browse/HDFS-9336
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9336.patch
>
>
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$DeleteSnapshotRequestProto$Builder.setSnapshotName(ClientNamenodeProtocolProtos.java:17509)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.deleteSnapshot(ClientNamenodeProtocolTranslatorPB.java:1005)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:255)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy15.deleteSnapshot(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.deleteSnapshot(DFSClient.java:2106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1660)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.deleteSnapshot(DistributedFileSystem.java:1677)
> at
> org.apache.hadoop.hdfs.web.TestWebHDFS.testWebHdfsAllowandDisallowSnapshots(TestWebHDFS.java:380)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9289) check genStamp when complete file
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980535#comment-14980535 ] Daryn Sharp commented on HDFS-9289: --- I worked with Chang on this issue and can't think of a scenario in which it's legitimate for the client to misreport the genstamp - whether the pipeline was updated or not. Consider a more extreme case: The client wrote more data after the pipeline recovered and misreports the older genstamp. That's silent data corruption! I'd like to see an exception here rather than later. > check genStamp when complete file > - > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9338) in webhdfs Null pointer exception will be thrown when xattrname is not given
Jagadesh Kiran N created HDFS-9338:
--
Summary: in webhdfs Null pointer exception will be thrown when
xattrname is not given
Key: HDFS-9338
URL: https://issues.apache.org/jira/browse/HDFS-9338
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Jagadesh Kiran N
Assignee: Jagadesh Kiran N
{code}
curl -i -X PUT
"http://10.19.92.127:50070/webhdfs/v1/kiran/?op=REMOVEXATTR=;
or
curl -i -X PUT "http://10.19.92.127:50070/webhdfs/v1/kiran/?op=REMOVEXATTR;
{code}
{code}
{"RemoteException":{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException","message":"XAttr
name cannot be null."}}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9339) Extend full test of KMS ACLs
[ https://issues.apache.org/jira/browse/HDFS-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-9339: --- Attachment: HDFS-9339.001.patch > Extend full test of KMS ACLs > > > Key: HDFS-9339 > URL: https://issues.apache.org/jira/browse/HDFS-9339 > Project: Hadoop HDFS > Issue Type: Test > Components: HDFS >Affects Versions: 2.7.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: HDFS-9339.001.patch > > > HDFS-9295 adds an end-to-end test for KMS, but it is missing a dimension. > The tests added in that JIRA hold the configuration constant and test that > all operations succeed or fail as expected. More tests are needed that hold > the operation constant and test that all possible configurations cause the > operations to succeed or fail as expected. This JIRA is to add those tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9277) IOException "Unable to load OAuth2 connection factory." in TestWebHDFSOAuth2.listStatusReturnsAsExpected
[ https://issues.apache.org/jira/browse/HDFS-9277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980677#comment-14980677 ] Wei-Chiu Chuang commented on HDFS-9277: --- Some test failures are unrelated to this fix: The TestBalancerWithMultipleNameNodes.testBalancer failure has appeared several times previously The two test cases in TestFsck had NoClassDefFoundError exception The timeout failure in TestNodeCount.testNodeCount has appeared several times previously. The failure in TestDataNodeMultipleRegistrations.testDNWithInvalidStorageWithHA and TestWebHDFS.testNamenodeRestart may be related to rev1 patch, and I'll dig into it. > IOException "Unable to load OAuth2 connection factory." in > TestWebHDFSOAuth2.listStatusReturnsAsExpected > > > Key: HDFS-9277 > URL: https://issues.apache.org/jira/browse/HDFS-9277 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Attachments: HDFS-9277.001.patch > > > This test is failing consistently in Hadoop-hdfs-trunk and > Hadoop-hdfs-trunk-Java8 since September 22. > REGRESSION: > org.apache.hadoop.hdfs.web.TestWebHDFSOAuth2.listStatusReturnsAsExpected > Error Message: > Unable to load OAuth2 connection factory. > Stack Trace: > java.io.IOException: Unable to load OAuth2 connection factory. > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.(FileInputStream.java:146) > at > org.apache.hadoop.security.ssl.ReloadingX509TrustManager.loadTrustManager(ReloadingX509TrustManager.java:164) > at > org.apache.hadoop.security.ssl.ReloadingX509TrustManager.(ReloadingX509TrustManager.java:81) > at > org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:215) > at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:131) > at > org.apache.hadoop.hdfs.web.URLConnectionFactory.newSslConnConfigurator(URLConnectionFactory.java:135) > at > org.apache.hadoop.hdfs.web.URLConnectionFactory.newOAuth2URLConnectionFactory(URLConnectionFactory.java:110) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.initialize(WebHdfsFileSystem.java:158) > at > org.apache.hadoop.hdfs.web.TestWebHDFSOAuth2.listStatusReturnsAsExpected(TestWebHDFSOAuth2.java:147) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9339) Extend full test of KMS ACLs
Daniel Templeton created HDFS-9339: -- Summary: Extend full test of KMS ACLs Key: HDFS-9339 URL: https://issues.apache.org/jira/browse/HDFS-9339 Project: Hadoop HDFS Issue Type: Test Components: HDFS Affects Versions: 2.7.1 Reporter: Daniel Templeton Assignee: Daniel Templeton HDFS-9295 adds an end-to-end test for KMS, but it is missing a dimension. The tests added in that JIRA hold the configuration constant and test that all operations succeed or fail as expected. More tests are needed that hold the operation constant and test that all possible configurations cause the operations to succeed or fail as expected. This JIRA is to add those tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9309) Tests that use KeyStoreUtil must call KeyStoreUtil.cleanupSSLConfig()
[
https://issues.apache.org/jira/browse/HDFS-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980697#comment-14980697
]
Wei-Chiu Chuang commented on HDFS-9309:
---
Except for TestNodeCount.testNodeCount, all other test failures are related to
Sasl, and more specifically errored on serverKS.jks not found.
> Tests that use KeyStoreUtil must call KeyStoreUtil.cleanupSSLConfig()
> -
>
> Key: HDFS-9309
> URL: https://issues.apache.org/jira/browse/HDFS-9309
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-9309.001.patch
>
>
> When KeyStoreUtil.setupSSLConfig() is called, several files are created
> (ssl-server.xml, ssl-client.xml, trustKS.jks, clientKS.jks, serverKS.jks).
> However, if they are not deleted upon exit, weird thing can happen to any
> subsequent tests.
> For example, if ssl-client.xml is not delete, but trustKS.jks is deleted,
> TestWebHDFSOAuth2.listStatusReturnsAsExpected will fail with message:
> {noformat}
> java.io.IOException: Unable to load OAuth2 connection factory.
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.(FileInputStream.java:146)
> at
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.loadTrustManager(ReloadingX509TrustManager.java:164)
> at
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.(ReloadingX509TrustManager.java:81)
> at
> org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:215)
> at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:131)
> at
> org.apache.hadoop.hdfs.web.URLConnectionFactory.newSslConnConfigurator(URLConnectionFactory.java:138)
> at
> org.apache.hadoop.hdfs.web.URLConnectionFactory.newOAuth2URLConnectionFactory(URLConnectionFactory.java:112)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.initialize(WebHdfsFileSystem.java:163)
> at
> org.apache.hadoop.hdfs.web.TestWebHDFSOAuth2.listStatusReturnsAsExpected(TestWebHDFSOAuth2.java:147)
> {noformat}
> There are currently several tests that do not clean up:
> {noformat}
> 130 ✗ weichiu@weichiu ~/trunk (trunk) $ grep -rnw . -e
> 'KeyStoreTestUtil\.setupSSLConfig' | cut -d: -f1 |xargs grep -L
> "KeyStoreTestUtil\.cleanupSSLConfig"
> ./hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/webapp/TestTimelineWebServicesWithSSL.java
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferTestCase.java
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestSecureNNWithQJM.java
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRespectsBindHostKeys.java
> ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/TestHttpFSFWithSWebhdfsFileSystem.java
> {noformat}
> This JIRA is the effort to remove the bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9057) allow/disallow snapshots via webhdfs
[ https://issues.apache.org/jira/browse/HDFS-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980553#comment-14980553 ] Brahma Reddy Battula commented on HDFS-9057: Hmm.. Yes..It's strange.. > allow/disallow snapshots via webhdfs > > > Key: HDFS-9057 > URL: https://issues.apache.org/jira/browse/HDFS-9057 > Project: Hadoop HDFS > Issue Type: New Feature > Components: webhdfs >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Brahma Reddy Battula > Attachments: HDFS-9057-002.patch, HDFS-9057-003.patch, > HDFS-9057-004.patch, HDFS-9057.patch > > > We should be able to allow and disallow directories for snapshotting via > WebHDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9337) In webhdfs Nullpoint exception will be thrown in renamesnapshot when oldsnapshotname is not given
Jagadesh Kiran N created HDFS-9337:
--
Summary: In webhdfs Nullpoint exception will be thrown in
renamesnapshot when oldsnapshotname is not given
Key: HDFS-9337
URL: https://issues.apache.org/jira/browse/HDFS-9337
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Jagadesh Kiran N
Assignee: Jagadesh Kiran N
{code}
curl -i -X PUT
"http://10.19.92.127:50070/webhdfs/v1/kiran/sreenu?op=RENAMESNAPSHOT=SNAPSHOTNAME;
{code}
Null point exception will be thrown
{code}
{"RemoteException":{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException","message":null}}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9309) Tests that use KeyStoreUtil must call KeyStoreUtil.cleanupSSLConfig()
[
https://issues.apache.org/jira/browse/HDFS-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-9309:
--
Attachment: HDFS-9309.002.patch
Fixed a bug in Sasl test case. SSL configs should be removed upon exiting.
> Tests that use KeyStoreUtil must call KeyStoreUtil.cleanupSSLConfig()
> -
>
> Key: HDFS-9309
> URL: https://issues.apache.org/jira/browse/HDFS-9309
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-9309.001.patch, HDFS-9309.002.patch
>
>
> When KeyStoreUtil.setupSSLConfig() is called, several files are created
> (ssl-server.xml, ssl-client.xml, trustKS.jks, clientKS.jks, serverKS.jks).
> However, if they are not deleted upon exit, weird thing can happen to any
> subsequent tests.
> For example, if ssl-client.xml is not delete, but trustKS.jks is deleted,
> TestWebHDFSOAuth2.listStatusReturnsAsExpected will fail with message:
> {noformat}
> java.io.IOException: Unable to load OAuth2 connection factory.
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.(FileInputStream.java:146)
> at
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.loadTrustManager(ReloadingX509TrustManager.java:164)
> at
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.(ReloadingX509TrustManager.java:81)
> at
> org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:215)
> at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:131)
> at
> org.apache.hadoop.hdfs.web.URLConnectionFactory.newSslConnConfigurator(URLConnectionFactory.java:138)
> at
> org.apache.hadoop.hdfs.web.URLConnectionFactory.newOAuth2URLConnectionFactory(URLConnectionFactory.java:112)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.initialize(WebHdfsFileSystem.java:163)
> at
> org.apache.hadoop.hdfs.web.TestWebHDFSOAuth2.listStatusReturnsAsExpected(TestWebHDFSOAuth2.java:147)
> {noformat}
> There are currently several tests that do not clean up:
> {noformat}
> 130 ✗ weichiu@weichiu ~/trunk (trunk) $ grep -rnw . -e
> 'KeyStoreTestUtil\.setupSSLConfig' | cut -d: -f1 |xargs grep -L
> "KeyStoreTestUtil\.cleanupSSLConfig"
> ./hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/webapp/TestTimelineWebServicesWithSSL.java
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferTestCase.java
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestSecureNNWithQJM.java
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRespectsBindHostKeys.java
> ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/TestHttpFSFWithSWebhdfsFileSystem.java
> {noformat}
> This JIRA is the effort to remove the bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9336) deleteSnapshot throws NPE when snapshotname is null
[
https://issues.apache.org/jira/browse/HDFS-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980876#comment-14980876
]
Brahma Reddy Battula commented on HDFS-9336:
[~rakeshr] thanks for looking into this issue.. Updated the patch.. Used
{{GenericTestUtils.assertExceptionContains}} ...
> deleteSnapshot throws NPE when snapshotname is null
> ---
>
> Key: HDFS-9336
> URL: https://issues.apache.org/jira/browse/HDFS-9336
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9336-002.patch, HDFS-9336.patch
>
>
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$DeleteSnapshotRequestProto$Builder.setSnapshotName(ClientNamenodeProtocolProtos.java:17509)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.deleteSnapshot(ClientNamenodeProtocolTranslatorPB.java:1005)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:255)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy15.deleteSnapshot(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.deleteSnapshot(DFSClient.java:2106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1660)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.deleteSnapshot(DistributedFileSystem.java:1677)
> at
> org.apache.hadoop.hdfs.web.TestWebHDFS.testWebHdfsAllowandDisallowSnapshots(TestWebHDFS.java:380)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7984) webhdfs:// needs to support provided delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-7984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HeeSoo Kim updated HDFS-7984: - Attachment: HDFS-7984.004.patch > webhdfs:// needs to support provided delegation tokens > -- > > Key: HDFS-7984 > URL: https://issues.apache.org/jira/browse/HDFS-7984 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: HeeSoo Kim >Priority: Blocker > Attachments: HDFS-7984.001.patch, HDFS-7984.002.patch, > HDFS-7984.003.patch, HDFS-7984.004.patch, HDFS-7984.patch > > > When using the webhdfs:// filesystem (especially from distcp), we need the > ability to inject a delegation token rather than webhdfs initialize its own. > This would allow for cross-authentication-zone file system accesses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9336) deleteSnapshot throws NPE when snapshotname is null
[
https://issues.apache.org/jira/browse/HDFS-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-9336:
---
Attachment: HDFS-9336-002.patch
> deleteSnapshot throws NPE when snapshotname is null
> ---
>
> Key: HDFS-9336
> URL: https://issues.apache.org/jira/browse/HDFS-9336
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9336-002.patch, HDFS-9336.patch
>
>
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$DeleteSnapshotRequestProto$Builder.setSnapshotName(ClientNamenodeProtocolProtos.java:17509)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.deleteSnapshot(ClientNamenodeProtocolTranslatorPB.java:1005)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:255)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy15.deleteSnapshot(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.deleteSnapshot(DFSClient.java:2106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1660)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$37.doCall(DistributedFileSystem.java:1)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.deleteSnapshot(DistributedFileSystem.java:1677)
> at
> org.apache.hadoop.hdfs.web.TestWebHDFS.testWebHdfsAllowandDisallowSnapshots(TestWebHDFS.java:380)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9340) libhdfspp fails to compile after HDFS-9207
[
https://issues.apache.org/jira/browse/HDFS-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980873#comment-14980873
]
Hadoop QA commented on HDFS-9340:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 5m 49s
{color} | {color:red} root in HDFS-8707 failed. {color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m 21s
{color} | {color:red} hadoop-hdfs-client in HDFS-8707 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 6s
{color} | {color:red} hadoop-hdfs-client in HDFS-8707 failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 8s
{color} | {color:red} hadoop-hdfs-client in HDFS-8707 failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 7s
{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m 8s
{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 7s
{color} | {color:red} hadoop-hdfs-client in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 7s
{color} | {color:red} hadoop-hdfs-client in the patch failed with JDK
v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 19s
{color} | {color:red} Patch generated 425 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 34s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.0 Server=1.7.0
Image:test-patch-base-hadoop-date2015-10-29 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12769571/HDFS-9340.HDFS-8707.000.patch
|
| JIRA Issue | HDFS-9340 |
| Optional Tests | asflicense javac javadoc mvninstall unit xml |
| uname | Linux d767612bb0e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-c3a2069/precommit/personality/hadoop.sh
|
| git revision | HDFS-8707 / 07c904d |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_66
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/13279/artifact/patchprocess/branch-mvninstall-root.txt
|
| mvneclipse |
https://builds.apache.org/job/PreCommit-HDFS-Build/13279/artifact/patchprocess/branch-mvneclipse-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/13279/artifact/patchprocess/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdk1.8.0_66.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/13279/artifact/patchprocess/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdk1.7.0_79.txt
|
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/13279/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| mvneclipse |
https://builds.apache.org/job/PreCommit-HDFS-Build/13279/artifact/patchprocess/patch-mvneclipse-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/13279/artifact/patchprocess/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdk1.8.0_66.txt
|
| javadoc |
[jira] [Commented] (HDFS-9309) Tests that use KeyStoreUtil must call KeyStoreUtil.cleanupSSLConfig()
[
https://issues.apache.org/jira/browse/HDFS-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980884#comment-14980884
]
Wei-Chiu Chuang commented on HDFS-9309:
---
I just realized using System.getProperty("test.build.data") is discouraged in
HADOOP-9263, so I will update the fix later.
> Tests that use KeyStoreUtil must call KeyStoreUtil.cleanupSSLConfig()
> -
>
> Key: HDFS-9309
> URL: https://issues.apache.org/jira/browse/HDFS-9309
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-9309.001.patch, HDFS-9309.002.patch
>
>
> When KeyStoreUtil.setupSSLConfig() is called, several files are created
> (ssl-server.xml, ssl-client.xml, trustKS.jks, clientKS.jks, serverKS.jks).
> However, if they are not deleted upon exit, weird thing can happen to any
> subsequent tests.
> For example, if ssl-client.xml is not delete, but trustKS.jks is deleted,
> TestWebHDFSOAuth2.listStatusReturnsAsExpected will fail with message:
> {noformat}
> java.io.IOException: Unable to load OAuth2 connection factory.
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.(FileInputStream.java:146)
> at
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.loadTrustManager(ReloadingX509TrustManager.java:164)
> at
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.(ReloadingX509TrustManager.java:81)
> at
> org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:215)
> at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:131)
> at
> org.apache.hadoop.hdfs.web.URLConnectionFactory.newSslConnConfigurator(URLConnectionFactory.java:138)
> at
> org.apache.hadoop.hdfs.web.URLConnectionFactory.newOAuth2URLConnectionFactory(URLConnectionFactory.java:112)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.initialize(WebHdfsFileSystem.java:163)
> at
> org.apache.hadoop.hdfs.web.TestWebHDFSOAuth2.listStatusReturnsAsExpected(TestWebHDFSOAuth2.java:147)
> {noformat}
> There are currently several tests that do not clean up:
> {noformat}
> 130 ✗ weichiu@weichiu ~/trunk (trunk) $ grep -rnw . -e
> 'KeyStoreTestUtil\.setupSSLConfig' | cut -d: -f1 |xargs grep -L
> "KeyStoreTestUtil\.cleanupSSLConfig"
> ./hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/webapp/TestTimelineWebServicesWithSSL.java
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferTestCase.java
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestSecureNNWithQJM.java
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRespectsBindHostKeys.java
> ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/TestHttpFSFWithSWebhdfsFileSystem.java
> {noformat}
> This JIRA is the effort to remove the bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9340) libhdfspp fails to compile after HDFS-9207
[
https://issues.apache.org/jira/browse/HDFS-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-9340:
-
Attachment: HDFS-9340.HDFS-8707.000.patch
> libhdfspp fails to compile after HDFS-9207
> --
>
> Key: HDFS-9340
> URL: https://issues.apache.org/jira/browse/HDFS-9340
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-9340.HDFS-8707.000.patch
>
>
> After the refactor of HDFS-9207 the {{hadoop-hdfs-client}} module fails to
> compile as it invokes {{cmake}} against a directory that does not exist. It
> should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9340) libhdfspp fails to compile after HDFS-9207
[
https://issues.apache.org/jira/browse/HDFS-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-9340:
-
Status: Patch Available (was: Open)
> libhdfspp fails to compile after HDFS-9207
> --
>
> Key: HDFS-9340
> URL: https://issues.apache.org/jira/browse/HDFS-9340
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-9340.HDFS-8707.000.patch
>
>
> After the refactor of HDFS-9207 the {{hadoop-hdfs-client}} module fails to
> compile as it invokes {{cmake}} against a directory that does not exist. It
> should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9236) Missing sanity check for block size during block recovery
[
https://issues.apache.org/jira/browse/HDFS-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980912#comment-14980912
]
Tony Wu commented on HDFS-9236:
---
[~yzhangal] Thanks a lot for looking at the patch.
> Missing sanity check for block size during block recovery
> -
>
> Key: HDFS-9236
> URL: https://issues.apache.org/jira/browse/HDFS-9236
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
> Attachments: HDFS-9236.001.patch, HDFS-9236.002.patch,
> HDFS-9236.003.patch
>
>
> Ran into an issue while running test against faulty data-node code.
> Currently in DataNode.java:
> {code:java}
> /** Block synchronization */
> void syncBlock(RecoveringBlock rBlock,
> List syncList) throws IOException {
> …
> // Calculate the best available replica state.
> ReplicaState bestState = ReplicaState.RWR;
> …
> // Calculate list of nodes that will participate in the recovery
> // and the new block size
> List participatingList = new ArrayList();
> final ExtendedBlock newBlock = new ExtendedBlock(bpid, blockId,
> -1, recoveryId);
> switch(bestState) {
> …
> case RBW:
> case RWR:
> long minLength = Long.MAX_VALUE;
> for(BlockRecord r : syncList) {
> ReplicaState rState = r.rInfo.getOriginalReplicaState();
> if(rState == bestState) {
> minLength = Math.min(minLength, r.rInfo.getNumBytes());
> participatingList.add(r);
> }
> }
> newBlock.setNumBytes(minLength);
> break;
> …
> }
> …
> nn.commitBlockSynchronization(block,
> newBlock.getGenerationStamp(), newBlock.getNumBytes(), true, false,
> datanodes, storages);
> }
> {code}
> This code is called by the DN coordinating the block recovery. In the above
> case, it is possible for none of the rState (reported by DNs with copies of
> the replica being recovered) to match the bestState. This can either be
> caused by faulty DN code or stale/modified/corrupted files on DN. When this
> happens the DN will end up reporting the minLengh of Long.MAX_VALUE.
> Unfortunately there is no check on the NN for replica length. See
> FSNamesystem.java:
> {code:java}
> void commitBlockSynchronization(ExtendedBlock oldBlock,
> long newgenerationstamp, long newlength,
> boolean closeFile, boolean deleteblock, DatanodeID[] newtargets,
> String[] newtargetstorages) throws IOException {
> …
> if (deleteblock) {
> Block blockToDel = ExtendedBlock.getLocalBlock(oldBlock);
> boolean remove = iFile.removeLastBlock(blockToDel) != null;
> if (remove) {
> blockManager.removeBlock(storedBlock);
> }
> } else {
> // update last block
> if(!copyTruncate) {
> storedBlock.setGenerationStamp(newgenerationstamp);
>
> // XXX block length is updated without any check <<< storedBlock.setNumBytes(newlength);
> }
> …
> if (closeFile) {
> LOG.info("commitBlockSynchronization(oldBlock=" + oldBlock
> + ", file=" + src
> + (copyTruncate ? ", newBlock=" + truncatedBlock
> : ", newgenerationstamp=" + newgenerationstamp)
> + ", newlength=" + newlength
> + ", newtargets=" + Arrays.asList(newtargets) + ") successful");
> } else {
> LOG.info("commitBlockSynchronization(" + oldBlock + ") successful");
> }
> }
> {code}
> After this point the block length becomes Long.MAX_VALUE. Any subsequent
> block report (even with correct length) will cause the block to be marked as
> corrupted. Since this is block could be the last block of the file. If this
> happens and the client goes away, NN won’t be able to recover the lease and
> close the file because the last block is under-replicated.
> I believe we need to have a sanity check for block size on both DN and NN to
> prevent such case from happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode
[
https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980760#comment-14980760
]
Hadoop QA commented on HDFS-9276:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 6s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 43s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 33s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 4s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 41s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 2s
{color} | {color:red} hadoop-common-project/hadoop-common introduced 1 new
FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 6s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 27s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 48s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 15s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 46s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 21s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 187m 55s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| | org.apache.hadoop.security.token.Token$PrivateToken doesn't override
Token.equals(Object) At Token.java:At Token.java:[line 1] |
| JDK v1.7.0_79 Failed junit tests | hadoop.ipc.TestRPC |
| | hadoop.ha.TestZKFailoverController |
| | hadoop.hdfs.server.datanode.TestFsDatasetCache |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| |
[jira] [Created] (HDFS-9340) libhdfspp fails to compile after HDFS-9207
Haohui Mai created HDFS-9340:
Summary: libhdfspp fails to compile after HDFS-9207
Key: HDFS-9340
URL: https://issues.apache.org/jira/browse/HDFS-9340
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
After the refactor of HDFS-9207 the {{hadoop-hdfs-client}} module fails to
compile as it invokes {{cmake}} against a directory that does not exist. It
should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9339) Extend full test of KMS ACLs
[
https://issues.apache.org/jira/browse/HDFS-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980906#comment-14980906
]
Zhe Zhang commented on HDFS-9339:
-
Thanks Daniel! The patch looks pretty good. A few minor comments:
# Maybe we should improve the criteria in assertions?
{code}
try {
setup(conf);
assertTrue("Exception during key creation with correct config"
+ " using whitelist key ACLs", createKey(realUgi, KEY1, conf));
} finally {
teardown();
}
{code}
The test will pass if {{createKey}} throws an exception. A simple fix is to
{{fail()}} in the {{finally}} statement. But maybe there's a way to avoid
adding it in every {{finally}} statement.
# Since the above {{try-finally}} logic repeats many times we can also abstract
it out as a method.
# Looks like the {{// Correct config with blacklist}} {{//Missing GET_METADATA
KMS ACL}} sections are both repeated twice?
> Extend full test of KMS ACLs
>
>
> Key: HDFS-9339
> URL: https://issues.apache.org/jira/browse/HDFS-9339
> Project: Hadoop HDFS
> Issue Type: Test
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: HDFS-9339.001.patch
>
>
> HDFS-9295 adds an end-to-end test for KMS, but it is missing a dimension.
> The tests added in that JIRA hold the configuration constant and test that
> all operations succeed or fail as expected. More tests are needed that hold
> the operation constant and test that all possible configurations cause the
> operations to succeed or fail as expected. This JIRA is to add those tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9236) Missing sanity check for block size during block recovery
[
https://issues.apache.org/jira/browse/HDFS-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980907#comment-14980907
]
Yongjun Zhang commented on HDFS-9236:
-
Sorry for the delay [~e90tony]. I did a review and I'm +1 on rev 003, will
commit soon.
> Missing sanity check for block size during block recovery
> -
>
> Key: HDFS-9236
> URL: https://issues.apache.org/jira/browse/HDFS-9236
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
> Attachments: HDFS-9236.001.patch, HDFS-9236.002.patch,
> HDFS-9236.003.patch
>
>
> Ran into an issue while running test against faulty data-node code.
> Currently in DataNode.java:
> {code:java}
> /** Block synchronization */
> void syncBlock(RecoveringBlock rBlock,
> List syncList) throws IOException {
> …
> // Calculate the best available replica state.
> ReplicaState bestState = ReplicaState.RWR;
> …
> // Calculate list of nodes that will participate in the recovery
> // and the new block size
> List participatingList = new ArrayList();
> final ExtendedBlock newBlock = new ExtendedBlock(bpid, blockId,
> -1, recoveryId);
> switch(bestState) {
> …
> case RBW:
> case RWR:
> long minLength = Long.MAX_VALUE;
> for(BlockRecord r : syncList) {
> ReplicaState rState = r.rInfo.getOriginalReplicaState();
> if(rState == bestState) {
> minLength = Math.min(minLength, r.rInfo.getNumBytes());
> participatingList.add(r);
> }
> }
> newBlock.setNumBytes(minLength);
> break;
> …
> }
> …
> nn.commitBlockSynchronization(block,
> newBlock.getGenerationStamp(), newBlock.getNumBytes(), true, false,
> datanodes, storages);
> }
> {code}
> This code is called by the DN coordinating the block recovery. In the above
> case, it is possible for none of the rState (reported by DNs with copies of
> the replica being recovered) to match the bestState. This can either be
> caused by faulty DN code or stale/modified/corrupted files on DN. When this
> happens the DN will end up reporting the minLengh of Long.MAX_VALUE.
> Unfortunately there is no check on the NN for replica length. See
> FSNamesystem.java:
> {code:java}
> void commitBlockSynchronization(ExtendedBlock oldBlock,
> long newgenerationstamp, long newlength,
> boolean closeFile, boolean deleteblock, DatanodeID[] newtargets,
> String[] newtargetstorages) throws IOException {
> …
> if (deleteblock) {
> Block blockToDel = ExtendedBlock.getLocalBlock(oldBlock);
> boolean remove = iFile.removeLastBlock(blockToDel) != null;
> if (remove) {
> blockManager.removeBlock(storedBlock);
> }
> } else {
> // update last block
> if(!copyTruncate) {
> storedBlock.setGenerationStamp(newgenerationstamp);
>
> // XXX block length is updated without any check <<< storedBlock.setNumBytes(newlength);
> }
> …
> if (closeFile) {
> LOG.info("commitBlockSynchronization(oldBlock=" + oldBlock
> + ", file=" + src
> + (copyTruncate ? ", newBlock=" + truncatedBlock
> : ", newgenerationstamp=" + newgenerationstamp)
> + ", newlength=" + newlength
> + ", newtargets=" + Arrays.asList(newtargets) + ") successful");
> } else {
> LOG.info("commitBlockSynchronization(" + oldBlock + ") successful");
> }
> }
> {code}
> After this point the block length becomes Long.MAX_VALUE. Any subsequent
> block report (even with correct length) will cause the block to be marked as
> corrupted. Since this is block could be the last block of the file. If this
> happens and the client goes away, NN won’t be able to recover the lease and
> close the file because the last block is under-replicated.
> I believe we need to have a sanity check for block size on both DN and NN to
> prevent such case from happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9333) Some tests using MiniDFSCluster errored complaining port in use
[
https://issues.apache.org/jira/browse/HDFS-9333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980975#comment-14980975
]
Masatake Iwasaki commented on HDFS-9333:
Thanks for reporting this, [~drankye].
TestBlockTokenWithDFSStriped already uses MiniDFSCluster with random port
settings and 49333 is the randomly chosen port. It could be race with another
process using ephemeral port.
TestDFSZKFailoverController seems to use fixed port settings because zkfc part
does not allow random port setting. Let me see this if you do not yet started
the work.
> Some tests using MiniDFSCluster errored complaining port in use
> ---
>
> Key: HDFS-9333
> URL: https://issues.apache.org/jira/browse/HDFS-9333
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Kai Zheng
>Priority: Minor
>
> Ref. the following:
> {noformat}
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.483 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
> testRead(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped)
> Time elapsed: 11.021 sec <<< ERROR!
> java.net.BindException: Port in use: localhost:49333
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> at
> org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:884)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:826)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:821)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:675)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2015)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1996)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS.doTestRead(TestBlockTokenWithDFS.java:539)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped.testRead(TestBlockTokenWithDFSStriped.java:62)
> {noformat}
> Another one:
> {noformat}
> Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 9.859 sec <<<
> FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController
> testFailoverAndBackOnNNShutdown(org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController)
> Time elapsed: 0.41 sec <<< ERROR!
> java.net.BindException: Problem binding to [localhost:10021]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:399)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1245)
> at
>
[jira] [Updated] (HDFS-9332) Fix Precondition failures from NameNodeEditLogRoller while saving namespace
[ https://issues.apache.org/jira/browse/HDFS-9332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-9332: -- Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Fixed, thanks for reviewing Yongjun and Mingliang! > Fix Precondition failures from NameNodeEditLogRoller while saving namespace > --- > > Key: HDFS-9332 > URL: https://issues.apache.org/jira/browse/HDFS-9332 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0 > > Attachments: HDFS-9332.001.patch > > > The check for the # of txns in the open edit log does not first check that an > edit log segment is open, leading to a Precondition failure. This surfaced at > HDFS-7871 which fixed that it was printing in a tight loop, but the cause of > the Precondition failure is still present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9236) Missing sanity check for block size during block recovery
[
https://issues.apache.org/jira/browse/HDFS-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981071#comment-14981071
]
Tony Wu commented on HDFS-9236:
---
Hi [~yzhangal], I believe HDFS-9255 has moved block recovery related code to a
different location. I will rebase my patch and upload a new one shortly.
> Missing sanity check for block size during block recovery
> -
>
> Key: HDFS-9236
> URL: https://issues.apache.org/jira/browse/HDFS-9236
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
> Attachments: HDFS-9236.001.patch, HDFS-9236.002.patch,
> HDFS-9236.003.patch
>
>
> Ran into an issue while running test against faulty data-node code.
> Currently in DataNode.java:
> {code:java}
> /** Block synchronization */
> void syncBlock(RecoveringBlock rBlock,
> List syncList) throws IOException {
> …
> // Calculate the best available replica state.
> ReplicaState bestState = ReplicaState.RWR;
> …
> // Calculate list of nodes that will participate in the recovery
> // and the new block size
> List participatingList = new ArrayList();
> final ExtendedBlock newBlock = new ExtendedBlock(bpid, blockId,
> -1, recoveryId);
> switch(bestState) {
> …
> case RBW:
> case RWR:
> long minLength = Long.MAX_VALUE;
> for(BlockRecord r : syncList) {
> ReplicaState rState = r.rInfo.getOriginalReplicaState();
> if(rState == bestState) {
> minLength = Math.min(minLength, r.rInfo.getNumBytes());
> participatingList.add(r);
> }
> }
> newBlock.setNumBytes(minLength);
> break;
> …
> }
> …
> nn.commitBlockSynchronization(block,
> newBlock.getGenerationStamp(), newBlock.getNumBytes(), true, false,
> datanodes, storages);
> }
> {code}
> This code is called by the DN coordinating the block recovery. In the above
> case, it is possible for none of the rState (reported by DNs with copies of
> the replica being recovered) to match the bestState. This can either be
> caused by faulty DN code or stale/modified/corrupted files on DN. When this
> happens the DN will end up reporting the minLengh of Long.MAX_VALUE.
> Unfortunately there is no check on the NN for replica length. See
> FSNamesystem.java:
> {code:java}
> void commitBlockSynchronization(ExtendedBlock oldBlock,
> long newgenerationstamp, long newlength,
> boolean closeFile, boolean deleteblock, DatanodeID[] newtargets,
> String[] newtargetstorages) throws IOException {
> …
> if (deleteblock) {
> Block blockToDel = ExtendedBlock.getLocalBlock(oldBlock);
> boolean remove = iFile.removeLastBlock(blockToDel) != null;
> if (remove) {
> blockManager.removeBlock(storedBlock);
> }
> } else {
> // update last block
> if(!copyTruncate) {
> storedBlock.setGenerationStamp(newgenerationstamp);
>
> // XXX block length is updated without any check <<< storedBlock.setNumBytes(newlength);
> }
> …
> if (closeFile) {
> LOG.info("commitBlockSynchronization(oldBlock=" + oldBlock
> + ", file=" + src
> + (copyTruncate ? ", newBlock=" + truncatedBlock
> : ", newgenerationstamp=" + newgenerationstamp)
> + ", newlength=" + newlength
> + ", newtargets=" + Arrays.asList(newtargets) + ") successful");
> } else {
> LOG.info("commitBlockSynchronization(" + oldBlock + ") successful");
> }
> }
> {code}
> After this point the block length becomes Long.MAX_VALUE. Any subsequent
> block report (even with correct length) will cause the block to be marked as
> corrupted. Since this is block could be the last block of the file. If this
> happens and the client goes away, NN won’t be able to recover the lease and
> close the file because the last block is under-replicated.
> I believe we need to have a sanity check for block size on both DN and NN to
> prevent such case from happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-9333) Some tests using MiniDFSCluster errored complaining port in use
[
https://issues.apache.org/jira/browse/HDFS-9333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki reassigned HDFS-9333:
--
Assignee: Masatake Iwasaki
> Some tests using MiniDFSCluster errored complaining port in use
> ---
>
> Key: HDFS-9333
> URL: https://issues.apache.org/jira/browse/HDFS-9333
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Kai Zheng
>Assignee: Masatake Iwasaki
>Priority: Minor
>
> Ref. the following:
> {noformat}
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.483 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
> testRead(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped)
> Time elapsed: 11.021 sec <<< ERROR!
> java.net.BindException: Port in use: localhost:49333
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> at
> org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:884)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:826)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:821)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:675)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2015)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1996)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS.doTestRead(TestBlockTokenWithDFS.java:539)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped.testRead(TestBlockTokenWithDFSStriped.java:62)
> {noformat}
> Another one:
> {noformat}
> Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 9.859 sec <<<
> FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController
> testFailoverAndBackOnNNShutdown(org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController)
> Time elapsed: 0.41 sec <<< ERROR!
> java.net.BindException: Problem binding to [localhost:10021]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:399)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1245)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1014)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:889)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:821)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:480)
> at
>
[jira] [Commented] (HDFS-9332) Fix Precondition failures from NameNodeEditLogRoller while saving namespace
[ https://issues.apache.org/jira/browse/HDFS-9332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980991#comment-14980991 ] Andrew Wang commented on HDFS-9332: --- Failed tests are unrelated, ran them locally. I also filed YETUS-146 for what look like some Yetus display bugs. Will commit shortly. > Fix Precondition failures from NameNodeEditLogRoller while saving namespace > --- > > Key: HDFS-9332 > URL: https://issues.apache.org/jira/browse/HDFS-9332 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-9332.001.patch > > > The check for the # of txns in the open edit log does not first check that an > edit log segment is open, leading to a Precondition failure. This surfaced at > HDFS-7871 which fixed that it was printing in a tight loop, but the cause of > the Precondition failure is still present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8777) Erasure Coding: add tests for taking snapshots on EC files
[
https://issues.apache.org/jira/browse/HDFS-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981077#comment-14981077
]
Zhe Zhang commented on HDFS-8777:
-
Thanks Rakesh for the patch! It looks good overall. A few minor comments:
# Maybe explicitly use {{sysDefaultPolicy}} in all {{setErasureCodingPolicy}}
calls in this test? Now there's a mix of {{sysDefaultPolicy}} and null.
# Missing "coding": "// Now delete the dir which has erasure policy."
# In section "// Check that older snapshots still have the old ECPolicy
settings", should we also verify {{snap2}}?
# The below verifies the EC policy on a non-existent file. It looks like a bug
in {{getErasureCodingPolicy}}. Actually it looks like
{{getEncryptionZoneForPath}} has the same behavior.
{code}
final Path snap1File = new Path(snap1, "file1");
assertEquals("Got unexpected erasure coding policy", sysDefaultPolicy,
fs.getErasureCodingPolicy(snap1File));
{code}
# As a follow-on, we can explore whether it's possible to subclass
{{TestSnapshot}} with {{setErasureCodingPolicy}} hooks. I'm not sure how hard
it is, just a thought.
> Erasure Coding: add tests for taking snapshots on EC files
> --
>
> Key: HDFS-8777
> URL: https://issues.apache.org/jira/browse/HDFS-8777
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Jing Zhao
>Assignee: Rakesh R
> Labels: test
> Attachments: HDFS-8777-01.patch, HDFS-8777-02.patch,
> HDFS-8777-HDFS-7285-00.patch, HDFS-8777-HDFS-7285-01.patch
>
>
> We need to add more tests for (EC + snapshots). The tests need to verify the
> fsimage saving/loading is correct.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9340) libhdfspp fails to compile after HDFS-9207
[
https://issues.apache.org/jira/browse/HDFS-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981103#comment-14981103
]
Hadoop QA commented on HDFS-9340:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
38s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 24s
{color} | {color:red} Patch generated 425 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 34s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-10-29 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12769571/HDFS-9340.HDFS-8707.000.patch
|
| JIRA Issue | HDFS-9340 |
| Optional Tests | asflicense javac javadoc mvninstall unit xml |
| uname | Linux 54caa9d90fe5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build@2/patchprocess/apache-yetus-c3a2069/precommit/personality/hadoop.sh
|
| git revision | HDFS-8707 / 85232c6 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_60
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 |
| JDK v1.7.0_79 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13283/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDFS-Build/13283/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Max memory used | 230MB |
| Powered by | Apache Yetus http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13283/console |
This message was automatically generated.
> libhdfspp fails to compile after HDFS-9207
> --
>
> Key: HDFS-9340
> URL: https://issues.apache.org/jira/browse/HDFS-9340
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-9340.HDFS-8707.000.patch
>
>
> After the refactor of HDFS-9207 the {{hadoop-hdfs-client}} module fails to
> compile as it invokes {{cmake}} against a directory that does not exist. It
> should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9309) Tests that use KeyStoreUtil must call KeyStoreUtil.cleanupSSLConfig()
[
https://issues.apache.org/jira/browse/HDFS-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981166#comment-14981166
]
Hadoop QA commented on HDFS-9309:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 51s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 28s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
58s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 0s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 58s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 44s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 57s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 44s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 34s
{color} | {color:green} hadoop-kms in the patch passed with JDK v1.8.0_60.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 50s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 23s
{color} | {color:green} hadoop-hdfs-httpfs in the patch passed with JDK
v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 4s
{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the
patch passed with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 38s
{color} | {color:green} hadoop-kms in the patch passed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 54m 33s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 30s
{color} | {color:green} hadoop-hdfs-httpfs in the patch passed with JDK
v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 17s
{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the
patch passed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 21s
{color} | {color:red}
[jira] [Updated] (HDFS-9229) Expose size of NameNode directory as a metric
[ https://issues.apache.org/jira/browse/HDFS-9229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-9229: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Thanks Surendra! +1 on the latest patch. I just committed it to trunk and branch-2. Also thanks Rakesh, Haohui, and Jing for the helpful reviews. > Expose size of NameNode directory as a metric > - > > Key: HDFS-9229 > URL: https://issues.apache.org/jira/browse/HDFS-9229 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Surendra Singh Lilhore >Priority: Minor > Fix For: 2.8.0 > > Attachments: HDFS-9229.001.patch, HDFS-9229.002.patch, > HDFS-9229.003.patch, HDFS-9229.004.patch, HDFS-9229.005.patch, > HDFS-9229.006.patch > > > Useful for admins in reserving / managing NN local file system space. Also > useful when transferring NN backups. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9236) Missing sanity check for block size during block recovery
[
https://issues.apache.org/jira/browse/HDFS-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981025#comment-14981025
]
Yongjun Zhang commented on HDFS-9236:
-
Sorry [~twu], the patch no longer applies because of other commits. would you
please update the patch? thanks.
> Missing sanity check for block size during block recovery
> -
>
> Key: HDFS-9236
> URL: https://issues.apache.org/jira/browse/HDFS-9236
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
> Attachments: HDFS-9236.001.patch, HDFS-9236.002.patch,
> HDFS-9236.003.patch
>
>
> Ran into an issue while running test against faulty data-node code.
> Currently in DataNode.java:
> {code:java}
> /** Block synchronization */
> void syncBlock(RecoveringBlock rBlock,
> List syncList) throws IOException {
> …
> // Calculate the best available replica state.
> ReplicaState bestState = ReplicaState.RWR;
> …
> // Calculate list of nodes that will participate in the recovery
> // and the new block size
> List participatingList = new ArrayList();
> final ExtendedBlock newBlock = new ExtendedBlock(bpid, blockId,
> -1, recoveryId);
> switch(bestState) {
> …
> case RBW:
> case RWR:
> long minLength = Long.MAX_VALUE;
> for(BlockRecord r : syncList) {
> ReplicaState rState = r.rInfo.getOriginalReplicaState();
> if(rState == bestState) {
> minLength = Math.min(minLength, r.rInfo.getNumBytes());
> participatingList.add(r);
> }
> }
> newBlock.setNumBytes(minLength);
> break;
> …
> }
> …
> nn.commitBlockSynchronization(block,
> newBlock.getGenerationStamp(), newBlock.getNumBytes(), true, false,
> datanodes, storages);
> }
> {code}
> This code is called by the DN coordinating the block recovery. In the above
> case, it is possible for none of the rState (reported by DNs with copies of
> the replica being recovered) to match the bestState. This can either be
> caused by faulty DN code or stale/modified/corrupted files on DN. When this
> happens the DN will end up reporting the minLengh of Long.MAX_VALUE.
> Unfortunately there is no check on the NN for replica length. See
> FSNamesystem.java:
> {code:java}
> void commitBlockSynchronization(ExtendedBlock oldBlock,
> long newgenerationstamp, long newlength,
> boolean closeFile, boolean deleteblock, DatanodeID[] newtargets,
> String[] newtargetstorages) throws IOException {
> …
> if (deleteblock) {
> Block blockToDel = ExtendedBlock.getLocalBlock(oldBlock);
> boolean remove = iFile.removeLastBlock(blockToDel) != null;
> if (remove) {
> blockManager.removeBlock(storedBlock);
> }
> } else {
> // update last block
> if(!copyTruncate) {
> storedBlock.setGenerationStamp(newgenerationstamp);
>
> // XXX block length is updated without any check <<< storedBlock.setNumBytes(newlength);
> }
> …
> if (closeFile) {
> LOG.info("commitBlockSynchronization(oldBlock=" + oldBlock
> + ", file=" + src
> + (copyTruncate ? ", newBlock=" + truncatedBlock
> : ", newgenerationstamp=" + newgenerationstamp)
> + ", newlength=" + newlength
> + ", newtargets=" + Arrays.asList(newtargets) + ") successful");
> } else {
> LOG.info("commitBlockSynchronization(" + oldBlock + ") successful");
> }
> }
> {code}
> After this point the block length becomes Long.MAX_VALUE. Any subsequent
> block report (even with correct length) will cause the block to be marked as
> corrupted. Since this is block could be the last block of the file. If this
> happens and the client goes away, NN won’t be able to recover the lease and
> close the file because the last block is under-replicated.
> I believe we need to have a sanity check for block size on both DN and NN to
> prevent such case from happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9332) Fix Precondition failures from NameNodeEditLogRoller while saving namespace
[ https://issues.apache.org/jira/browse/HDFS-9332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981109#comment-14981109 ] Hudson commented on HDFS-9332: -- FAILURE: Integrated in Hadoop-trunk-Commit #8725 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8725/]) HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Fix Precondition failures from NameNodeEditLogRoller while saving namespace > --- > > Key: HDFS-9332 > URL: https://issues.apache.org/jira/browse/HDFS-9332 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0 > > Attachments: HDFS-9332.001.patch > > > The check for the # of txns in the open edit log does not first check that an > edit log segment is open, leading to a Precondition failure. This surfaced at > HDFS-7871 which fixed that it was printing in a tight loop, but the cause of > the Precondition failure is still present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9229) Expose size of NameNode directory as a metric
[ https://issues.apache.org/jira/browse/HDFS-9229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981108#comment-14981108 ] Hudson commented on HDFS-9229: -- FAILURE: Integrated in Hadoop-trunk-Commit #8725 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8725/]) HDFS-9229. Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java > Expose size of NameNode directory as a metric > - > > Key: HDFS-9229 > URL: https://issues.apache.org/jira/browse/HDFS-9229 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Surendra Singh Lilhore >Priority: Minor > Fix For: 2.8.0 > > Attachments: HDFS-9229.001.patch, HDFS-9229.002.patch, > HDFS-9229.003.patch, HDFS-9229.004.patch, HDFS-9229.005.patch, > HDFS-9229.006.patch > > > Useful for admins in reserving / managing NN local file system space. Also > useful when transferring NN backups. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-4937) ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom()
[
https://issues.apache.org/jira/browse/HDFS-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981132#comment-14981132
]
Kihwal Lee commented on HDFS-4937:
--
The failed test cases pass when run locally.
{noformat}
---
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 146.108 sec -
in org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.379 sec - in
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.582 sec - in
org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 104.369 sec -
in org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
Results :
Tests run: 54, Failures: 0, Errors: 0, Skipped: 0
{noformat}
Also, there actually is no new findbugs issue.
> ReplicationMonitor can infinite-loop in
> BlockPlacementPolicyDefault#chooseRandom()
> --
>
> Key: HDFS-4937
> URL: https://issues.apache.org/jira/browse/HDFS-4937
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.0.4-alpha, 0.23.8
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Labels: BB2015-05-TBR
> Attachments: HDFS-4937.patch, HDFS-4937.v1.patch
>
>
> When a large number of nodes are removed by refreshing node lists, the
> network topology is updated. If the refresh happens at the right moment, the
> replication monitor thread may stuck in the while loop of {{chooseRandom()}}.
> This is because the cached cluster size is used in the terminal condition
> check of the loop. This usually happens when a block with a high replication
> factor is being processed. Since replicas/rack is also calculated beforehand,
> no node choice may satisfy the goodness criteria if refreshing removed racks.
> All nodes will end up in the excluded list, but the size will still be less
> than the cached cluster size, so it will loop infinitely. This was observed
> in a production environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9332) Fix Precondition failures from NameNodeEditLogRoller while saving namespace
[ https://issues.apache.org/jira/browse/HDFS-9332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981160#comment-14981160 ] Hudson commented on HDFS-9332: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #602 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/602/]) HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Fix Precondition failures from NameNodeEditLogRoller while saving namespace > --- > > Key: HDFS-9332 > URL: https://issues.apache.org/jira/browse/HDFS-9332 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0 > > Attachments: HDFS-9332.001.patch > > > The check for the # of txns in the open edit log does not first check that an > edit log segment is open, leading to a Precondition failure. This surfaced at > HDFS-7871 which fixed that it was printing in a tight loop, but the cause of > the Precondition failure is still present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9229) Expose size of NameNode directory as a metric
[ https://issues.apache.org/jira/browse/HDFS-9229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981159#comment-14981159 ] Hudson commented on HDFS-9229: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #602 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/602/]) HDFS-9229. Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java > Expose size of NameNode directory as a metric > - > > Key: HDFS-9229 > URL: https://issues.apache.org/jira/browse/HDFS-9229 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Surendra Singh Lilhore >Priority: Minor > Fix For: 2.8.0 > > Attachments: HDFS-9229.001.patch, HDFS-9229.002.patch, > HDFS-9229.003.patch, HDFS-9229.004.patch, HDFS-9229.005.patch, > HDFS-9229.006.patch > > > Useful for admins in reserving / managing NN local file system space. Also > useful when transferring NN backups. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9336) deleteSnapshot throws NPE when snapshotname is null
[
https://issues.apache.org/jira/browse/HDFS-9336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981187#comment-14981187
]
Hadoop QA commented on HDFS-9336:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 5s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 6s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s
{color} | {color:red} Patch generated 1 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 57, now 57). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 54s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 15s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 14s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 156m 43s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_79 Failed junit tests |
hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
| | hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.0 Server=1.7.0
Image:test-patch-base-hadoop-date2015-10-29 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12769578/HDFS-9336-002.patch |
| JIRA Issue | HDFS-9336 |
| Optional Tests | asflicense javac javadoc mvninstall unit findbugs
checkstyle compile
[jira] [Commented] (HDFS-1477) Make NameNode Reconfigurable.
[
https://issues.apache.org/jira/browse/HDFS-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981231#comment-14981231
]
Arpit Agarwal commented on HDFS-1477:
-
Hi [~xiaobingo], thanks for taking this up. My comments on the v5 patch:
# FsNameSystem and DatanodeManager need implement Reconfigurable, since their
Reconfigurable methods are only invoked via NameNode.
# What does getNewConf() do? Looks like it just constructs a new config object
by re-reading the config files? There should have been a Javadoc on the base
class method, perhaps we can add one now.
# Can we just skip implementing reconfiguration via the servlet? A better
approach is to have the daemon re-read its config files like HDFS-6808 (which
you pointed out to me offline :-). Looks like you will need a new RPC call and
support on the client. Also it would be good to log the reconfiguration request
in the hdfs audit log.
# Nitpicks: Fix indentation in DatanodeManager#reconfigurePropertyImpl.
# {{#reconfigurePropertyImpl}} - get {{namesystem.writeLock()}} outside the
{{try}} block.
Other suggestions for separate subtasks.
# The list of reconfigurable properties should not be hard-coded. We can add an
annotation to the reconfigurable properties.
# Add {{ReconfigurableBase}} support for atomic updates, i.e. all or no changes
take effect.
# Documentation.
> Make NameNode Reconfigurable.
> -
>
> Key: HDFS-1477
> URL: https://issues.apache.org/jira/browse/HDFS-1477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Patrick Kling
>Assignee: Xiaobing Zhou
> Attachments: HDFS-1477.005.patch, HDFS-1477.2.patch,
> HDFS-1477.3.patch, HDFS-1477.4.patch, HDFS-1477.patch
>
>
> Modify NameNode to implement the interface Reconfigurable proposed in
> HADOOP-7001. This would allow us to change certain configuration properties
> without restarting the name node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9339) Extend full test of KMS ACLs
[ https://issues.apache.org/jira/browse/HDFS-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981239#comment-14981239 ] Daniel Templeton commented on HDFS-9339: 1. The createKey method is a local method that wraps the KeyProvider method. One of the things it does is return false if there's an exception. 2. It would be possible to abstract it out, but it would involve another anonymous inner class to perform the operations. I'm not sure that's an improvement. That's the way it's done in TestKMS, but I don't like it. :) 3. They aren't actually repeated. There are two different controlling ACLs, so one tests one ACL, and the other tests the other. > Extend full test of KMS ACLs > > > Key: HDFS-9339 > URL: https://issues.apache.org/jira/browse/HDFS-9339 > Project: Hadoop HDFS > Issue Type: Test > Components: HDFS >Affects Versions: 2.7.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: HDFS-9339.001.patch > > > HDFS-9295 adds an end-to-end test for KMS, but it is missing a dimension. > The tests added in that JIRA hold the configuration constant and test that > all operations succeed or fail as expected. More tests are needed that hold > the operation constant and test that all possible configurations cause the > operations to succeed or fail as expected. This JIRA is to add those tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9332) Fix Precondition failures from NameNodeEditLogRoller while saving namespace
[ https://issues.apache.org/jira/browse/HDFS-9332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981254#comment-14981254 ] Hudson commented on HDFS-9332: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2544 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2544/]) HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java > Fix Precondition failures from NameNodeEditLogRoller while saving namespace > --- > > Key: HDFS-9332 > URL: https://issues.apache.org/jira/browse/HDFS-9332 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0 > > Attachments: HDFS-9332.001.patch > > > The check for the # of txns in the open edit log does not first check that an > edit log segment is open, leading to a Precondition failure. This surfaced at > HDFS-7871 which fixed that it was printing in a tight loop, but the cause of > the Precondition failure is still present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7984) webhdfs:// needs to support provided delegation tokens
[
https://issues.apache.org/jira/browse/HDFS-7984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981275#comment-14981275
]
Hadoop QA commented on HDFS-7984:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 48s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 9s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 51s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 51s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 3s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 47s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 45s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 46s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 3s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 25s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 20s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 204m 40s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_79 Failed junit tests | hadoop.metrics2.impl.TestMetricsSystemImpl
|
| | hadoop.test.TestTimedOutTestsListener |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| |
[jira] [Commented] (HDFS-9236) Missing sanity check for block size during block recovery
[
https://issues.apache.org/jira/browse/HDFS-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981330#comment-14981330
]
Mingliang Liu commented on HDFS-9236:
-
The latest patch looks good to me overall. One minor comment: is it possible to
assert expected exception thrown (e.g. by error message) ?
> Missing sanity check for block size during block recovery
> -
>
> Key: HDFS-9236
> URL: https://issues.apache.org/jira/browse/HDFS-9236
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
> Attachments: HDFS-9236.001.patch, HDFS-9236.002.patch,
> HDFS-9236.003.patch, HDFS-9236.004.patch
>
>
> Ran into an issue while running test against faulty data-node code.
> Currently in DataNode.java:
> {code:java}
> /** Block synchronization */
> void syncBlock(RecoveringBlock rBlock,
> List syncList) throws IOException {
> …
> // Calculate the best available replica state.
> ReplicaState bestState = ReplicaState.RWR;
> …
> // Calculate list of nodes that will participate in the recovery
> // and the new block size
> List participatingList = new ArrayList();
> final ExtendedBlock newBlock = new ExtendedBlock(bpid, blockId,
> -1, recoveryId);
> switch(bestState) {
> …
> case RBW:
> case RWR:
> long minLength = Long.MAX_VALUE;
> for(BlockRecord r : syncList) {
> ReplicaState rState = r.rInfo.getOriginalReplicaState();
> if(rState == bestState) {
> minLength = Math.min(minLength, r.rInfo.getNumBytes());
> participatingList.add(r);
> }
> }
> newBlock.setNumBytes(minLength);
> break;
> …
> }
> …
> nn.commitBlockSynchronization(block,
> newBlock.getGenerationStamp(), newBlock.getNumBytes(), true, false,
> datanodes, storages);
> }
> {code}
> This code is called by the DN coordinating the block recovery. In the above
> case, it is possible for none of the rState (reported by DNs with copies of
> the replica being recovered) to match the bestState. This can either be
> caused by faulty DN code or stale/modified/corrupted files on DN. When this
> happens the DN will end up reporting the minLengh of Long.MAX_VALUE.
> Unfortunately there is no check on the NN for replica length. See
> FSNamesystem.java:
> {code:java}
> void commitBlockSynchronization(ExtendedBlock oldBlock,
> long newgenerationstamp, long newlength,
> boolean closeFile, boolean deleteblock, DatanodeID[] newtargets,
> String[] newtargetstorages) throws IOException {
> …
> if (deleteblock) {
> Block blockToDel = ExtendedBlock.getLocalBlock(oldBlock);
> boolean remove = iFile.removeLastBlock(blockToDel) != null;
> if (remove) {
> blockManager.removeBlock(storedBlock);
> }
> } else {
> // update last block
> if(!copyTruncate) {
> storedBlock.setGenerationStamp(newgenerationstamp);
>
> // XXX block length is updated without any check <<< storedBlock.setNumBytes(newlength);
> }
> …
> if (closeFile) {
> LOG.info("commitBlockSynchronization(oldBlock=" + oldBlock
> + ", file=" + src
> + (copyTruncate ? ", newBlock=" + truncatedBlock
> : ", newgenerationstamp=" + newgenerationstamp)
> + ", newlength=" + newlength
> + ", newtargets=" + Arrays.asList(newtargets) + ") successful");
> } else {
> LOG.info("commitBlockSynchronization(" + oldBlock + ") successful");
> }
> }
> {code}
> After this point the block length becomes Long.MAX_VALUE. Any subsequent
> block report (even with correct length) will cause the block to be marked as
> corrupted. Since this is block could be the last block of the file. If this
> happens and the client goes away, NN won’t be able to recover the lease and
> close the file because the last block is under-replicated.
> I believe we need to have a sanity check for block size on both DN and NN to
> prevent such case from happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9289) check genStamp when complete file
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981222#comment-14981222 ] Chang Li commented on HDFS-9289: Thanks [~jingzhao], [~zhz] and [~daryn] for reivew and valuable discussions! Some additional info about several cases of mismatched GS we encountered is that they all happened after pipelineupdate for datanode close recovery, so no mismatched size of commit happen only mismatched GS. Could we reach a consensus of whether we should log warn of mismatched GS block info or throw exception? > check genStamp when complete file > - > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9229) Expose size of NameNode directory as a metric
[ https://issues.apache.org/jira/browse/HDFS-9229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981253#comment-14981253 ] Hudson commented on HDFS-9229: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2544 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2544/]) HDFS-9229. Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java > Expose size of NameNode directory as a metric > - > > Key: HDFS-9229 > URL: https://issues.apache.org/jira/browse/HDFS-9229 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Surendra Singh Lilhore >Priority: Minor > Fix For: 2.8.0 > > Attachments: HDFS-9229.001.patch, HDFS-9229.002.patch, > HDFS-9229.003.patch, HDFS-9229.004.patch, HDFS-9229.005.patch, > HDFS-9229.006.patch > > > Useful for admins in reserving / managing NN local file system space. Also > useful when transferring NN backups. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9339) Extend full test of KMS ACLs
[ https://issues.apache.org/jira/browse/HDFS-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981299#comment-14981299 ] Daniel Templeton commented on HDFS-9339: bq. Maybe remove the try-finally and let the entire test fail if setup fails? If setup throws an exception, the exception isn't caught, so after executing the finally block, the exception will continue being passed up the stack, eventually causing the test to error. > Extend full test of KMS ACLs > > > Key: HDFS-9339 > URL: https://issues.apache.org/jira/browse/HDFS-9339 > Project: Hadoop HDFS > Issue Type: Test > Components: HDFS >Affects Versions: 2.7.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: HDFS-9339.001.patch > > > HDFS-9295 adds an end-to-end test for KMS, but it is missing a dimension. > The tests added in that JIRA hold the configuration constant and test that > all operations succeed or fail as expected. More tests are needed that hold > the operation constant and test that all possible configurations cause the > operations to succeed or fail as expected. This JIRA is to add those tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9079) Erasure coding: preallocate multiple generation stamps and serialize updates from data streamers
[
https://issues.apache.org/jira/browse/HDFS-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981316#comment-14981316
]
Hadoop QA commented on HDFS-9079:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 0s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 21s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 34s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 21s
{color} | {color:red} Patch generated 103 new checkstyle issues in
hadoop-hdfs-project (total was 366, now 461). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 46s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 51s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 53s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 21s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 145m 47s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_79 Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.TestMiniDFSCluster |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| |
[jira] [Commented] (HDFS-9340) libhdfspp fails to compile after HDFS-9207
[
https://issues.apache.org/jira/browse/HDFS-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981341#comment-14981341
]
Bob Hansen commented on HDFS-9340:
--
[~wheat9]: this removes the old and broken pom file; is there an analog that
_does_ compile and test the libhdfspp project? Should this have been moved
rather than deleted?
> libhdfspp fails to compile after HDFS-9207
> --
>
> Key: HDFS-9340
> URL: https://issues.apache.org/jira/browse/HDFS-9340
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-9340.HDFS-8707.000.patch
>
>
> After the refactor of HDFS-9207 the {{hadoop-hdfs-client}} module fails to
> compile as it invokes {{cmake}} against a directory that does not exist. It
> should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9289) check genStamp when complete file
[
https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981262#comment-14981262
]
Zhe Zhang commented on HDFS-9289:
-
bq. That's silent data corruption!
[~daryn] I agree it's a silent data corruption in the current logic because we
update the NN's copy of the GS with the reported GS from the client:
{code}
// BlockInfo#commitBlock
this.set(getBlockId(), block.getNumBytes(), block.getGenerationStamp());
{code}
Throwing an exception (and therefore denying the commitBlock) turns this into
an explicit failure, which is better. But it's still a data loss because the
data written by the client after {{updatePipeline}} becomes invisible.
So I think at least for this particular bug (lacking {{volatile}}), the right
thing to do is to avoid changing NN's copy of GS when committing block (so we
should avoid changing blockID as well). The only thing we should commit is
{{numBytes}}. Of course we should still print a {{WARN}} or {{ERROR}} when GSes
mismatch. As a safer first step we should at least avoid decrementing NN's copy
of block GS.
In general, if a client misreports GS, does it indicate a likelihood of
misreported {{numBytes}} -- and therefore we should deny the {{commitBlock}}?
It's hard to say; the {{volatile}} bug here is only for GS. But since we have
already ensured the NN's copy of block {{numBytes}} never decrements, the harm
of a misreported {{numBytes}} is not severe.
> check genStamp when complete file
> -
>
> Key: HDFS-9289
> URL: https://issues.apache.org/jira/browse/HDFS-9289
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
>Priority: Critical
> Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch,
> HDFS-9289.4.patch
>
>
> we have seen a case of corrupt block which is caused by file complete after a
> pipelineUpdate, but the file complete with the old block genStamp. This
> caused the replicas of two datanodes in updated pipeline to be viewed as
> corrupte. Propose to check genstamp when commit block
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9339) Extend full test of KMS ACLs
[
https://issues.apache.org/jira/browse/HDFS-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981286#comment-14981286
]
Zhe Zhang commented on HDFS-9339:
-
bq. The createKey method is a local method that wraps the KeyProvider method.
One of the things it does is return false if there's an exception.
I see, so we {{try}} for {{setup}}. But should we consider the test pass if
{{setup}} fails from an exception (and therefore the {{assertTrue}} is not
visited)? Maybe remove the {{try-finally}} and let the entire test fail if
{{setup}} fails?
bq. It would be possible to abstract it out, but it would involve another
anonymous inner class
I see, thanks for explaining.
bq. They aren't actually repeated.
My bad, spotted the differences now.
+1 pending a conclusion on the first point.
> Extend full test of KMS ACLs
>
>
> Key: HDFS-9339
> URL: https://issues.apache.org/jira/browse/HDFS-9339
> Project: Hadoop HDFS
> Issue Type: Test
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: HDFS-9339.001.patch
>
>
> HDFS-9295 adds an end-to-end test for KMS, but it is missing a dimension.
> The tests added in that JIRA hold the configuration constant and test that
> all operations succeed or fail as expected. More tests are needed that hold
> the operation constant and test that all possible configurations cause the
> operations to succeed or fail as expected. This JIRA is to add those tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9340) libhdfspp fails to compile after HDFS-9207
[
https://issues.apache.org/jira/browse/HDFS-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981295#comment-14981295
]
Haohui Mai commented on HDFS-9340:
--
I propose fixing the issues of ASF licensing header in a separate jira.
[~bobhansen] and [~James Clampffer], can you please review? Thanks.
> libhdfspp fails to compile after HDFS-9207
> --
>
> Key: HDFS-9340
> URL: https://issues.apache.org/jira/browse/HDFS-9340
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-9340.HDFS-8707.000.patch
>
>
> After the refactor of HDFS-9207 the {{hadoop-hdfs-client}} module fails to
> compile as it invokes {{cmake}} against a directory that does not exist. It
> should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9289) check genStamp when complete file
[
https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981313#comment-14981313
]
Zhe Zhang commented on HDFS-9289:
-
Thanks Jing for the explanation. I agree it's reasonable to throw an exception
in {{commitBlock}} and rely on lease recovery to bring the block back to full
strength in this case.
> check genStamp when complete file
> -
>
> Key: HDFS-9289
> URL: https://issues.apache.org/jira/browse/HDFS-9289
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
>Priority: Critical
> Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch,
> HDFS-9289.4.patch
>
>
> we have seen a case of corrupt block which is caused by file complete after a
> pipelineUpdate, but the file complete with the old block genStamp. This
> caused the replicas of two datanodes in updated pipeline to be viewed as
> corrupte. Propose to check genstamp when commit block
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7964) Add support for async edit logging
[ https://issues.apache.org/jira/browse/HDFS-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981323#comment-14981323 ] Daryn Sharp commented on HDFS-7964: --- I think special casing a few ops is fragile even if we add an assert. I feel safer knowing a production deadlock isn't going to occur if/when someone doesn't add tests to stress all possible code paths or more importantly doesn't test against the async logging. A runtime check will require halting the NN because data structures are already updated - definitely not good when a simple sync check will prevent any issues. Exploring latency concerns: Today, logEdit has to serialize and compute the crc for ops while holding the write lock. With this feature, logEdit only queues which releases the write lock sooner. The serialization cost is shifted to the background thread, however there's not 100 threads contending. In the meantime other calls are being processed. Sadly, the slow steady stream concern isn't even possible. Even with 20-30k ops/sec, the average batch size is 1.6. I've seen as high as 7. Now once fine-grain locking is done... We may be able to have that debate. ;) > Add support for async edit logging > -- > > Key: HDFS-7964 > URL: https://issues.apache.org/jira/browse/HDFS-7964 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.0.2-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-7964.patch, HDFS-7964.patch, HDFS-7964.patch > > > Edit logging is a major source of contention within the NN. LogEdit is > called within the namespace write log, while logSync is called outside of the > lock to allow greater concurrency. The handler thread remains busy until > logSync returns to provide the client with a durability guarantee for the > response. > Write heavy RPC load and/or slow IO causes handlers to stall in logSync. > Although the write lock is not held, readers are limited/starved and the call > queue fills. Combining an edit log thread with postponed RPC responses from > HADOOP-10300 will provide the same durability guarantee but immediately free > up the handlers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) check genStamp when complete file
[
https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981321#comment-14981321
]
Zhe Zhang commented on HDFS-9289:
-
A small ask for the next rev:
{code}
// BlockInfo#commitBlock
- this.set(getBlockId(), block.getNumBytes(), block.getGenerationStamp());
+ this.setNumBytes(block.getNumBytes());
{code}
We also need to add a test in the 04 patch. Otherwise LGTM.
> check genStamp when complete file
> -
>
> Key: HDFS-9289
> URL: https://issues.apache.org/jira/browse/HDFS-9289
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
>Priority: Critical
> Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch,
> HDFS-9289.4.patch
>
>
> we have seen a case of corrupt block which is caused by file complete after a
> pipelineUpdate, but the file complete with the old block genStamp. This
> caused the replicas of two datanodes in updated pipeline to be viewed as
> corrupte. Propose to check genstamp when commit block
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9236) Missing sanity check for block size during block recovery
[
https://issues.apache.org/jira/browse/HDFS-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tony Wu updated HDFS-9236:
--
Attachment: HDFS-9236.004.patch
In v4 patch:
* Rebased to latest trunk (moved the changes to new file
{{BlockRecoveryWorker.java}}).
> Missing sanity check for block size during block recovery
> -
>
> Key: HDFS-9236
> URL: https://issues.apache.org/jira/browse/HDFS-9236
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
> Attachments: HDFS-9236.001.patch, HDFS-9236.002.patch,
> HDFS-9236.003.patch, HDFS-9236.004.patch
>
>
> Ran into an issue while running test against faulty data-node code.
> Currently in DataNode.java:
> {code:java}
> /** Block synchronization */
> void syncBlock(RecoveringBlock rBlock,
> List syncList) throws IOException {
> …
> // Calculate the best available replica state.
> ReplicaState bestState = ReplicaState.RWR;
> …
> // Calculate list of nodes that will participate in the recovery
> // and the new block size
> List participatingList = new ArrayList();
> final ExtendedBlock newBlock = new ExtendedBlock(bpid, blockId,
> -1, recoveryId);
> switch(bestState) {
> …
> case RBW:
> case RWR:
> long minLength = Long.MAX_VALUE;
> for(BlockRecord r : syncList) {
> ReplicaState rState = r.rInfo.getOriginalReplicaState();
> if(rState == bestState) {
> minLength = Math.min(minLength, r.rInfo.getNumBytes());
> participatingList.add(r);
> }
> }
> newBlock.setNumBytes(minLength);
> break;
> …
> }
> …
> nn.commitBlockSynchronization(block,
> newBlock.getGenerationStamp(), newBlock.getNumBytes(), true, false,
> datanodes, storages);
> }
> {code}
> This code is called by the DN coordinating the block recovery. In the above
> case, it is possible for none of the rState (reported by DNs with copies of
> the replica being recovered) to match the bestState. This can either be
> caused by faulty DN code or stale/modified/corrupted files on DN. When this
> happens the DN will end up reporting the minLengh of Long.MAX_VALUE.
> Unfortunately there is no check on the NN for replica length. See
> FSNamesystem.java:
> {code:java}
> void commitBlockSynchronization(ExtendedBlock oldBlock,
> long newgenerationstamp, long newlength,
> boolean closeFile, boolean deleteblock, DatanodeID[] newtargets,
> String[] newtargetstorages) throws IOException {
> …
> if (deleteblock) {
> Block blockToDel = ExtendedBlock.getLocalBlock(oldBlock);
> boolean remove = iFile.removeLastBlock(blockToDel) != null;
> if (remove) {
> blockManager.removeBlock(storedBlock);
> }
> } else {
> // update last block
> if(!copyTruncate) {
> storedBlock.setGenerationStamp(newgenerationstamp);
>
> // XXX block length is updated without any check <<< storedBlock.setNumBytes(newlength);
> }
> …
> if (closeFile) {
> LOG.info("commitBlockSynchronization(oldBlock=" + oldBlock
> + ", file=" + src
> + (copyTruncate ? ", newBlock=" + truncatedBlock
> : ", newgenerationstamp=" + newgenerationstamp)
> + ", newlength=" + newlength
> + ", newtargets=" + Arrays.asList(newtargets) + ") successful");
> } else {
> LOG.info("commitBlockSynchronization(" + oldBlock + ") successful");
> }
> }
> {code}
> After this point the block length becomes Long.MAX_VALUE. Any subsequent
> block report (even with correct length) will cause the block to be marked as
> corrupted. Since this is block could be the last block of the file. If this
> happens and the client goes away, NN won’t be able to recover the lease and
> close the file because the last block is under-replicated.
> I believe we need to have a sanity check for block size on both DN and NN to
> prevent such case from happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-1477) Make NameNode Reconfigurable.
[
https://issues.apache.org/jira/browse/HDFS-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981231#comment-14981231
]
Arpit Agarwal edited comment on HDFS-1477 at 10/29/15 8:43 PM:
---
Hi [~xiaobingo], thanks for taking this up. My comments on the v5 patch:
# FsNameSystem and DatanodeManager need not implement Reconfigurable, since
their Reconfigurable methods are only invoked via NameNode.
# What does getNewConf() do? Looks like it just constructs a new config object
by re-reading the config files? There should have been a Javadoc on the base
class method, perhaps we can add one now.
# Can we just skip implementing reconfiguration via the servlet? A better
approach is to have the daemon re-read its config files like HDFS-6808 (which
you pointed out to me offline :-). Looks like you will need a new RPC call and
support on the client. Also it would be good to log the reconfiguration request
in the hdfs audit log.
# Nitpicks: Fix indentation in DatanodeManager#reconfigurePropertyImpl.
# {{#reconfigurePropertyImpl}} - get {{namesystem.writeLock()}} outside the
{{try}} block.
Other suggestions for separate subtasks.
# The list of reconfigurable properties should not be hard-coded. We can add an
annotation to the reconfigurable properties.
# Add {{ReconfigurableBase}} support for atomic updates, i.e. all or no changes
take effect.
# Documentation.
was (Author: arpitagarwal):
Hi [~xiaobingo], thanks for taking this up. My comments on the v5 patch:
# FsNameSystem and DatanodeManager need implement Reconfigurable, since their
Reconfigurable methods are only invoked via NameNode.
# What does getNewConf() do? Looks like it just constructs a new config object
by re-reading the config files? There should have been a Javadoc on the base
class method, perhaps we can add one now.
# Can we just skip implementing reconfiguration via the servlet? A better
approach is to have the daemon re-read its config files like HDFS-6808 (which
you pointed out to me offline :-). Looks like you will need a new RPC call and
support on the client. Also it would be good to log the reconfiguration request
in the hdfs audit log.
# Nitpicks: Fix indentation in DatanodeManager#reconfigurePropertyImpl.
# {{#reconfigurePropertyImpl}} - get {{namesystem.writeLock()}} outside the
{{try}} block.
Other suggestions for separate subtasks.
# The list of reconfigurable properties should not be hard-coded. We can add an
annotation to the reconfigurable properties.
# Add {{ReconfigurableBase}} support for atomic updates, i.e. all or no changes
take effect.
# Documentation.
> Make NameNode Reconfigurable.
> -
>
> Key: HDFS-1477
> URL: https://issues.apache.org/jira/browse/HDFS-1477
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Patrick Kling
>Assignee: Xiaobing Zhou
> Attachments: HDFS-1477.005.patch, HDFS-1477.2.patch,
> HDFS-1477.3.patch, HDFS-1477.4.patch, HDFS-1477.patch
>
>
> Modify NameNode to implement the interface Reconfigurable proposed in
> HADOOP-7001. This would allow us to change certain configuration properties
> without restarting the name node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9236) Missing sanity check for block size during block recovery
[
https://issues.apache.org/jira/browse/HDFS-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981248#comment-14981248
]
Yongjun Zhang commented on HDFS-9236:
-
Thanks [~twu], +1 on rev4 pending jenkins.
> Missing sanity check for block size during block recovery
> -
>
> Key: HDFS-9236
> URL: https://issues.apache.org/jira/browse/HDFS-9236
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
> Attachments: HDFS-9236.001.patch, HDFS-9236.002.patch,
> HDFS-9236.003.patch, HDFS-9236.004.patch
>
>
> Ran into an issue while running test against faulty data-node code.
> Currently in DataNode.java:
> {code:java}
> /** Block synchronization */
> void syncBlock(RecoveringBlock rBlock,
> List syncList) throws IOException {
> …
> // Calculate the best available replica state.
> ReplicaState bestState = ReplicaState.RWR;
> …
> // Calculate list of nodes that will participate in the recovery
> // and the new block size
> List participatingList = new ArrayList();
> final ExtendedBlock newBlock = new ExtendedBlock(bpid, blockId,
> -1, recoveryId);
> switch(bestState) {
> …
> case RBW:
> case RWR:
> long minLength = Long.MAX_VALUE;
> for(BlockRecord r : syncList) {
> ReplicaState rState = r.rInfo.getOriginalReplicaState();
> if(rState == bestState) {
> minLength = Math.min(minLength, r.rInfo.getNumBytes());
> participatingList.add(r);
> }
> }
> newBlock.setNumBytes(minLength);
> break;
> …
> }
> …
> nn.commitBlockSynchronization(block,
> newBlock.getGenerationStamp(), newBlock.getNumBytes(), true, false,
> datanodes, storages);
> }
> {code}
> This code is called by the DN coordinating the block recovery. In the above
> case, it is possible for none of the rState (reported by DNs with copies of
> the replica being recovered) to match the bestState. This can either be
> caused by faulty DN code or stale/modified/corrupted files on DN. When this
> happens the DN will end up reporting the minLengh of Long.MAX_VALUE.
> Unfortunately there is no check on the NN for replica length. See
> FSNamesystem.java:
> {code:java}
> void commitBlockSynchronization(ExtendedBlock oldBlock,
> long newgenerationstamp, long newlength,
> boolean closeFile, boolean deleteblock, DatanodeID[] newtargets,
> String[] newtargetstorages) throws IOException {
> …
> if (deleteblock) {
> Block blockToDel = ExtendedBlock.getLocalBlock(oldBlock);
> boolean remove = iFile.removeLastBlock(blockToDel) != null;
> if (remove) {
> blockManager.removeBlock(storedBlock);
> }
> } else {
> // update last block
> if(!copyTruncate) {
> storedBlock.setGenerationStamp(newgenerationstamp);
>
> // XXX block length is updated without any check <<< storedBlock.setNumBytes(newlength);
> }
> …
> if (closeFile) {
> LOG.info("commitBlockSynchronization(oldBlock=" + oldBlock
> + ", file=" + src
> + (copyTruncate ? ", newBlock=" + truncatedBlock
> : ", newgenerationstamp=" + newgenerationstamp)
> + ", newlength=" + newlength
> + ", newtargets=" + Arrays.asList(newtargets) + ") successful");
> } else {
> LOG.info("commitBlockSynchronization(" + oldBlock + ") successful");
> }
> }
> {code}
> After this point the block length becomes Long.MAX_VALUE. Any subsequent
> block report (even with correct length) will cause the block to be marked as
> corrupted. Since this is block could be the last block of the file. If this
> happens and the client goes away, NN won’t be able to recover the lease and
> close the file because the last block is under-replicated.
> I believe we need to have a sanity check for block size on both DN and NN to
> prevent such case from happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9289) check genStamp when complete file
[
https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981291#comment-14981291
]
Jing Zhao commented on HDFS-9289:
-
bq. In general, if a client misreports GS, does it indicate a likelihood of
misreported numBytes – and therefore we should deny the commitBlock?
Currently NN only depends on the reported length from the client to determine
the block length (not considering lease recovery scenario). So the only check
we can do about the length is the existing one: {{assert block.getNumBytes() <=
commitBlock.getNumBytes()}}.
bq. But it's still a data loss because the data written by the client after
updatePipeline becomes invisible.
Throwing an exception here may not necessarily mean that the data written after
updatePipeline will get lost. In most cases the data can still get recovered
during the lease recovery, considering the replicas have already get persisted
in DataNodes before client sends out the commit/complete request to NN (since
the client has received the last response from the pipeline at that time).
So throwing exception here should be the correct behavior and may not be that
risky.
> check genStamp when complete file
> -
>
> Key: HDFS-9289
> URL: https://issues.apache.org/jira/browse/HDFS-9289
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
>Priority: Critical
> Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch,
> HDFS-9289.4.patch
>
>
> we have seen a case of corrupt block which is caused by file complete after a
> pipelineUpdate, but the file complete with the old block genStamp. This
> caused the replicas of two datanodes in updated pipeline to be viewed as
> corrupte. Propose to check genstamp when commit block
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9340) libhdfspp fails to compile after HDFS-9207
[
https://issues.apache.org/jira/browse/HDFS-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981342#comment-14981342
]
Bob Hansen commented on HDFS-9340:
--
The ASF licensing headers can be taken care of as part of HDFS-9233.
> libhdfspp fails to compile after HDFS-9207
> --
>
> Key: HDFS-9340
> URL: https://issues.apache.org/jira/browse/HDFS-9340
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-9340.HDFS-8707.000.patch
>
>
> After the refactor of HDFS-9207 the {{hadoop-hdfs-client}} module fails to
> compile as it invokes {{cmake}} against a directory that does not exist. It
> should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9233) Create LICENSE.txt and NOTICES files for libhdfs++
[ https://issues.apache.org/jira/browse/HDFS-9233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981343#comment-14981343 ] Bob Hansen commented on HDFS-9233: -- Also: clean up ASF licensing checks in build > Create LICENSE.txt and NOTICES files for libhdfs++ > -- > > Key: HDFS-9233 > URL: https://issues.apache.org/jira/browse/HDFS-9233 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen > > We use third-party libraries that are Apache and Google licensed, and may be > adding an MIT-licenced third-party library. We need to include the > appropriate license files for inclusion into Apache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7984) webhdfs:// needs to support provided delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-7984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981358#comment-14981358 ] HeeSoo Kim commented on HDFS-7984: -- The test failures are unrelated to the change made for this jira. > webhdfs:// needs to support provided delegation tokens > -- > > Key: HDFS-7984 > URL: https://issues.apache.org/jira/browse/HDFS-7984 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: HeeSoo Kim >Priority: Blocker > Attachments: HDFS-7984.001.patch, HDFS-7984.002.patch, > HDFS-7984.003.patch, HDFS-7984.004.patch, HDFS-7984.patch > > > When using the webhdfs:// filesystem (especially from distcp), we need the > ability to inject a delegation token rather than webhdfs initialize its own. > This would allow for cross-authentication-zone file system accesses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9332) Fix Precondition failures from NameNodeEditLogRoller while saving namespace
[ https://issues.apache.org/jira/browse/HDFS-9332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981386#comment-14981386 ] Hudson commented on HDFS-9332: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1337 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1337/]) HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Fix Precondition failures from NameNodeEditLogRoller while saving namespace > --- > > Key: HDFS-9332 > URL: https://issues.apache.org/jira/browse/HDFS-9332 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0 > > Attachments: HDFS-9332.001.patch > > > The check for the # of txns in the open edit log does not first check that an > edit log segment is open, leading to a Precondition failure. This surfaced at > HDFS-7871 which fixed that it was printing in a tight loop, but the cause of > the Precondition failure is still present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9260) Improve performance and GC friendliness of startup and FBRs
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981442#comment-14981442 ] Staffan Friberg commented on HDFS-9260: --- I have been running through the other failed tests without being able to reproduce them locally. Was able to reproduce the failed test in hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes once on the trunk, but not yet with my branch. So it seems like these might be intermittent issues. > Improve performance and GC friendliness of startup and FBRs > --- > > Key: HDFS-9260 > URL: https://issues.apache.org/jira/browse/HDFS-9260 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg > Attachments: HDFS Block and Replica Management 20151013.pdf, > HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch, > HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch, > HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch > > > This patch changes the datastructures used for BlockInfos and Replicas to > keep them sorted. This allows faster and more GC friendly handling of full > block reports. > Would like to hear peoples feedback on this change and also some help > investigating/understanding a few outstanding issues if we are interested in > moving forward with this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9236) Missing sanity check for block size during block recovery
[
https://issues.apache.org/jira/browse/HDFS-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981458#comment-14981458
]
Mingliang Liu commented on HDFS-9236:
-
Sorry for the confusion.
By "assert expected exception thrown (e.g. by error message)", I mean
{{asserTrue(ioe.getMessage().contains("ooxx"));}} in test, not in the DN code.
I'm with you. I believe throwing an exception is correct and assert is wrong in
this case.
> Missing sanity check for block size during block recovery
> -
>
> Key: HDFS-9236
> URL: https://issues.apache.org/jira/browse/HDFS-9236
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
> Attachments: HDFS-9236.001.patch, HDFS-9236.002.patch,
> HDFS-9236.003.patch, HDFS-9236.004.patch
>
>
> Ran into an issue while running test against faulty data-node code.
> Currently in DataNode.java:
> {code:java}
> /** Block synchronization */
> void syncBlock(RecoveringBlock rBlock,
> List syncList) throws IOException {
> …
> // Calculate the best available replica state.
> ReplicaState bestState = ReplicaState.RWR;
> …
> // Calculate list of nodes that will participate in the recovery
> // and the new block size
> List participatingList = new ArrayList();
> final ExtendedBlock newBlock = new ExtendedBlock(bpid, blockId,
> -1, recoveryId);
> switch(bestState) {
> …
> case RBW:
> case RWR:
> long minLength = Long.MAX_VALUE;
> for(BlockRecord r : syncList) {
> ReplicaState rState = r.rInfo.getOriginalReplicaState();
> if(rState == bestState) {
> minLength = Math.min(minLength, r.rInfo.getNumBytes());
> participatingList.add(r);
> }
> }
> newBlock.setNumBytes(minLength);
> break;
> …
> }
> …
> nn.commitBlockSynchronization(block,
> newBlock.getGenerationStamp(), newBlock.getNumBytes(), true, false,
> datanodes, storages);
> }
> {code}
> This code is called by the DN coordinating the block recovery. In the above
> case, it is possible for none of the rState (reported by DNs with copies of
> the replica being recovered) to match the bestState. This can either be
> caused by faulty DN code or stale/modified/corrupted files on DN. When this
> happens the DN will end up reporting the minLengh of Long.MAX_VALUE.
> Unfortunately there is no check on the NN for replica length. See
> FSNamesystem.java:
> {code:java}
> void commitBlockSynchronization(ExtendedBlock oldBlock,
> long newgenerationstamp, long newlength,
> boolean closeFile, boolean deleteblock, DatanodeID[] newtargets,
> String[] newtargetstorages) throws IOException {
> …
> if (deleteblock) {
> Block blockToDel = ExtendedBlock.getLocalBlock(oldBlock);
> boolean remove = iFile.removeLastBlock(blockToDel) != null;
> if (remove) {
> blockManager.removeBlock(storedBlock);
> }
> } else {
> // update last block
> if(!copyTruncate) {
> storedBlock.setGenerationStamp(newgenerationstamp);
>
> // XXX block length is updated without any check <<< storedBlock.setNumBytes(newlength);
> }
> …
> if (closeFile) {
> LOG.info("commitBlockSynchronization(oldBlock=" + oldBlock
> + ", file=" + src
> + (copyTruncate ? ", newBlock=" + truncatedBlock
> : ", newgenerationstamp=" + newgenerationstamp)
> + ", newlength=" + newlength
> + ", newtargets=" + Arrays.asList(newtargets) + ") successful");
> } else {
> LOG.info("commitBlockSynchronization(" + oldBlock + ") successful");
> }
> }
> {code}
> After this point the block length becomes Long.MAX_VALUE. Any subsequent
> block report (even with correct length) will cause the block to be marked as
> corrupted. Since this is block could be the last block of the file. If this
> happens and the client goes away, NN won’t be able to recover the lease and
> close the file because the last block is under-replicated.
> I believe we need to have a sanity check for block size on both DN and NN to
> prevent such case from happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9340) libhdfspp fails to compile after HDFS-9207
[
https://issues.apache.org/jira/browse/HDFS-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981461#comment-14981461
]
James Clampffer commented on HDFS-9340:
---
Fantastic. It looks like it's building all the binaries. What was the old
directory it was trying to compile? libhdfspp/src/main/native in the old
location?
Haohui, is there a flag I can use with maven to run tests on just the native
code? I'd like to verify all our gmock tests are running before I +1 but it's
taking a while because I'm running all tests (in a VM) at the moment.
Also Is there a good reference on the build and test systems, ideally including
the implementation? The set of flags for maven you gave me worked but I
haven't found much documentation other than the very basic stuff in "How To
Contribute" and the pages that links to. If there aren't any docs and I have
to learn from scratch is there anywhere in the tree that you think would be
best to start looking? Thanks
> libhdfspp fails to compile after HDFS-9207
> --
>
> Key: HDFS-9340
> URL: https://issues.apache.org/jira/browse/HDFS-9340
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-9340.HDFS-8707.000.patch
>
>
> After the refactor of HDFS-9207 the {{hadoop-hdfs-client}} module fails to
> compile as it invokes {{cmake}} against a directory that does not exist. It
> should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9229) Expose size of NameNode directory as a metric
[ https://issues.apache.org/jira/browse/HDFS-9229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981385#comment-14981385 ] Hudson commented on HDFS-9229: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1337 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1337/]) HDFS-9229. Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java > Expose size of NameNode directory as a metric > - > > Key: HDFS-9229 > URL: https://issues.apache.org/jira/browse/HDFS-9229 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Surendra Singh Lilhore >Priority: Minor > Fix For: 2.8.0 > > Attachments: HDFS-9229.001.patch, HDFS-9229.002.patch, > HDFS-9229.003.patch, HDFS-9229.004.patch, HDFS-9229.005.patch, > HDFS-9229.006.patch > > > Useful for admins in reserving / managing NN local file system space. Also > useful when transferring NN backups. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9332) Fix Precondition failures from NameNodeEditLogRoller while saving namespace
[ https://issues.apache.org/jira/browse/HDFS-9332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981414#comment-14981414 ] Hudson commented on HDFS-9332: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #614 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/614/]) HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java > Fix Precondition failures from NameNodeEditLogRoller while saving namespace > --- > > Key: HDFS-9332 > URL: https://issues.apache.org/jira/browse/HDFS-9332 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0 > > Attachments: HDFS-9332.001.patch > > > The check for the # of txns in the open edit log does not first check that an > edit log segment is open, leading to a Precondition failure. This surfaced at > HDFS-7871 which fixed that it was printing in a tight loop, but the cause of > the Precondition failure is still present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9229) Expose size of NameNode directory as a metric
[ https://issues.apache.org/jira/browse/HDFS-9229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981413#comment-14981413 ] Hudson commented on HDFS-9229: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #614 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/614/]) HDFS-9229. Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md > Expose size of NameNode directory as a metric > - > > Key: HDFS-9229 > URL: https://issues.apache.org/jira/browse/HDFS-9229 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.1 >Reporter: Zhe Zhang >Assignee: Surendra Singh Lilhore >Priority: Minor > Fix For: 2.8.0 > > Attachments: HDFS-9229.001.patch, HDFS-9229.002.patch, > HDFS-9229.003.patch, HDFS-9229.004.patch, HDFS-9229.005.patch, > HDFS-9229.006.patch > > > Useful for admins in reserving / managing NN local file system space. Also > useful when transferring NN backups. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9289) check genStamp when complete file
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated HDFS-9289: --- Attachment: HDFS-9289.6.patch decide to not add new exception, update .6 patch > check genStamp when complete file > - > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9341) Simulated slow disk in SimulatedFSDataset
Zhe Zhang created HDFS-9341:
---
Summary: Simulated slow disk in SimulatedFSDataset
Key: HDFS-9341
URL: https://issues.apache.org/jira/browse/HDFS-9341
Project: Hadoop HDFS
Issue Type: Test
Components: test
Affects Versions: 2.7.1
Reporter: Zhe Zhang
Priority: Minor
Besides simulating the byte content, {{SimulatedFSDataset}} can also simulate
the scenario where disk is slow when accessing certain bytes. The slowness can
be random or controlled at certain bytes. It can also be made configurable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9129) Move the safemode block count into BlockManager
[
https://issues.apache.org/jira/browse/HDFS-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9129:
Attachment: HDFS-9129.016.patch
The failing tests can pass on my local machine. The v16 patch is to address the
findbugs warnings.
> Move the safemode block count into BlockManager
> ---
>
> Key: HDFS-9129
> URL: https://issues.apache.org/jira/browse/HDFS-9129
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Mingliang Liu
> Attachments: HDFS-9129.000.patch, HDFS-9129.001.patch,
> HDFS-9129.002.patch, HDFS-9129.003.patch, HDFS-9129.004.patch,
> HDFS-9129.005.patch, HDFS-9129.006.patch, HDFS-9129.007.patch,
> HDFS-9129.008.patch, HDFS-9129.009.patch, HDFS-9129.010.patch,
> HDFS-9129.011.patch, HDFS-9129.012.patch, HDFS-9129.013.patch,
> HDFS-9129.014.patch, HDFS-9129.015.patch, HDFS-9129.016.patch
>
>
> The {{SafeMode}} needs to track whether there are enough blocks so that the
> NN can get out of the safemode. These fields can moved to the
> {{BlockManager}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9340) libhdfspp fails to compile after HDFS-9207
[
https://issues.apache.org/jira/browse/HDFS-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981513#comment-14981513
]
Haohui Mai commented on HDFS-9340:
--
bq. Fantastic. It looks like it's building all the binaries. What was the old
directory it was trying to compile? libhdfspp/src/main/native in the old
location?
Yes. Looks like there is a merge error happened during HDFS-9170 / HDFS-9207.
bq. Haohui, is there a flag I can use with maven to run tests on just the
native code? I'd like to verify all our gmock tests are running before I +1 but
it's taking a while because I'm running all tests (in a VM) at the moment.
Not yet -- the current pom.xml does not run {{make test}}. Will need to file
another jira to fix it.
bq. Also Is there a good reference on the build and test systems, ideally
including the implementation? The set of flags for maven you gave me worked but
I haven't found much documentation other than the very basic stuff in "How To
Contribute" and the pages that links to. If there aren't any docs and I have to
learn from scratch is there anywhere in the tree that you think would be best
to start looking? Thanks
Can you be more concrete on what kinds of documentation can help here? It will
be great to capture your experience in the wiki for other contributors.
> libhdfspp fails to compile after HDFS-9207
> --
>
> Key: HDFS-9340
> URL: https://issues.apache.org/jira/browse/HDFS-9340
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-9340.HDFS-8707.000.patch
>
>
> After the refactor of HDFS-9207 the {{hadoop-hdfs-client}} module fails to
> compile as it invokes {{cmake}} against a directory that does not exist. It
> should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9339) Extend full test of KMS ACLs
[
https://issues.apache.org/jira/browse/HDFS-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981349#comment-14981349
]
Zhe Zhang commented on HDFS-9339:
-
Ah thanks, my bad. Then maybe we should {{try-finally}} for the entire test?
> Extend full test of KMS ACLs
>
>
> Key: HDFS-9339
> URL: https://issues.apache.org/jira/browse/HDFS-9339
> Project: Hadoop HDFS
> Issue Type: Test
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: HDFS-9339.001.patch
>
>
> HDFS-9295 adds an end-to-end test for KMS, but it is missing a dimension.
> The tests added in that JIRA hold the configuration constant and test that
> all operations succeed or fail as expected. More tests are needed that hold
> the operation constant and test that all possible configurations cause the
> operations to succeed or fail as expected. This JIRA is to add those tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-4937) ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom()
[
https://issues.apache.org/jira/browse/HDFS-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-4937:
-
Attachment: HDFS-4937.v2.patch
Daryn and I were looking at the patch and realized it can be improved.
Attaching a new patch.
> ReplicationMonitor can infinite-loop in
> BlockPlacementPolicyDefault#chooseRandom()
> --
>
> Key: HDFS-4937
> URL: https://issues.apache.org/jira/browse/HDFS-4937
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.0.4-alpha, 0.23.8
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Labels: BB2015-05-TBR
> Attachments: HDFS-4937.patch, HDFS-4937.v1.patch, HDFS-4937.v2.patch
>
>
> When a large number of nodes are removed by refreshing node lists, the
> network topology is updated. If the refresh happens at the right moment, the
> replication monitor thread may stuck in the while loop of {{chooseRandom()}}.
> This is because the cached cluster size is used in the terminal condition
> check of the loop. This usually happens when a block with a high replication
> factor is being processed. Since replicas/rack is also calculated beforehand,
> no node choice may satisfy the goodness criteria if refreshing removed racks.
> All nodes will end up in the excluded list, but the size will still be less
> than the cached cluster size, so it will loop infinitely. This was observed
> in a production environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9236) Missing sanity check for block size during block recovery
[
https://issues.apache.org/jira/browse/HDFS-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981419#comment-14981419
]
Tony Wu commented on HDFS-9236:
---
Hi [~liuml07],
Thanks a lot for your comment. I debated about having an assert as well and
think it has a few disadvantages (please correct me if I'm wrong):
# Assert can be disabled at runtime.
# Assert message is only visible on DN where the exception can propagate back
to NN (and also visible on DN).
# Assert would have stopped the DN process, which seems to be an overkill.
Given these reasons I think throwing an exception is the better choice.
Tony
> Missing sanity check for block size during block recovery
> -
>
> Key: HDFS-9236
> URL: https://issues.apache.org/jira/browse/HDFS-9236
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
> Attachments: HDFS-9236.001.patch, HDFS-9236.002.patch,
> HDFS-9236.003.patch, HDFS-9236.004.patch
>
>
> Ran into an issue while running test against faulty data-node code.
> Currently in DataNode.java:
> {code:java}
> /** Block synchronization */
> void syncBlock(RecoveringBlock rBlock,
> List syncList) throws IOException {
> …
> // Calculate the best available replica state.
> ReplicaState bestState = ReplicaState.RWR;
> …
> // Calculate list of nodes that will participate in the recovery
> // and the new block size
> List participatingList = new ArrayList();
> final ExtendedBlock newBlock = new ExtendedBlock(bpid, blockId,
> -1, recoveryId);
> switch(bestState) {
> …
> case RBW:
> case RWR:
> long minLength = Long.MAX_VALUE;
> for(BlockRecord r : syncList) {
> ReplicaState rState = r.rInfo.getOriginalReplicaState();
> if(rState == bestState) {
> minLength = Math.min(minLength, r.rInfo.getNumBytes());
> participatingList.add(r);
> }
> }
> newBlock.setNumBytes(minLength);
> break;
> …
> }
> …
> nn.commitBlockSynchronization(block,
> newBlock.getGenerationStamp(), newBlock.getNumBytes(), true, false,
> datanodes, storages);
> }
> {code}
> This code is called by the DN coordinating the block recovery. In the above
> case, it is possible for none of the rState (reported by DNs with copies of
> the replica being recovered) to match the bestState. This can either be
> caused by faulty DN code or stale/modified/corrupted files on DN. When this
> happens the DN will end up reporting the minLengh of Long.MAX_VALUE.
> Unfortunately there is no check on the NN for replica length. See
> FSNamesystem.java:
> {code:java}
> void commitBlockSynchronization(ExtendedBlock oldBlock,
> long newgenerationstamp, long newlength,
> boolean closeFile, boolean deleteblock, DatanodeID[] newtargets,
> String[] newtargetstorages) throws IOException {
> …
> if (deleteblock) {
> Block blockToDel = ExtendedBlock.getLocalBlock(oldBlock);
> boolean remove = iFile.removeLastBlock(blockToDel) != null;
> if (remove) {
> blockManager.removeBlock(storedBlock);
> }
> } else {
> // update last block
> if(!copyTruncate) {
> storedBlock.setGenerationStamp(newgenerationstamp);
>
> // XXX block length is updated without any check <<< storedBlock.setNumBytes(newlength);
> }
> …
> if (closeFile) {
> LOG.info("commitBlockSynchronization(oldBlock=" + oldBlock
> + ", file=" + src
> + (copyTruncate ? ", newBlock=" + truncatedBlock
> : ", newgenerationstamp=" + newgenerationstamp)
> + ", newlength=" + newlength
> + ", newtargets=" + Arrays.asList(newtargets) + ") successful");
> } else {
> LOG.info("commitBlockSynchronization(" + oldBlock + ") successful");
> }
> }
> {code}
> After this point the block length becomes Long.MAX_VALUE. Any subsequent
> block report (even with correct length) will cause the block to be marked as
> corrupted. Since this is block could be the last block of the file. If this
> happens and the client goes away, NN won’t be able to recover the lease and
> close the file because the last block is under-replicated.
> I believe we need to have a sanity check for block size on both DN and NN to
> prevent such case from happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9289) check genStamp when complete file
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated HDFS-9289: --- Attachment: HDFS-9289.5.patch Thanks [~jingzhao] and [~zhz] for further review! Uploaded .5 patch which not only log detailed error message but also throw exception. Also commit does not modify NN's copy of the block GS as Zhang suggested. Test is included. > check genStamp when complete file > - > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9282) Make data directory count and storage raw capacity related tests FsDataset-agnostic
[ https://issues.apache.org/jira/browse/HDFS-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Wu updated HDFS-9282: -- Attachment: HDFS-9282.003.patch In v3 patch: * Rebased patch on latest trunk. > Make data directory count and storage raw capacity related tests > FsDataset-agnostic > --- > > Key: HDFS-9282 > URL: https://issues.apache.org/jira/browse/HDFS-9282 > Project: Hadoop HDFS > Issue Type: Improvement > Components: HDFS, test >Affects Versions: 2.7.1 >Reporter: Tony Wu >Assignee: Tony Wu >Priority: Minor > Attachments: HDFS-9282.001.patch, HDFS-9282.002.patch, > HDFS-9282.003.patch > > > DFSMiniCluster and several tests have hard coded assumption of the underlying > storage having 2 data directories (volumes). As HDFS-9188 pointed out, with > new FsDataset implementations, these hard coded assumption about number of > data directories and raw capacities of storage may change as well. > We need to extend FsDatasetTestUtils to provide: > * Number of data directories of underlying storage per DataNode > * Raw storage capacity of underlying storage per DataNode. > * Have MiniDFSCluster automatically pick up the correct values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HDFS-9263:
--
Target Version/s: 3.0.0, 2.6.2, 2.7.3
We should backport this to maintenance releases too to keep the test-runs in
sync.
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9236) Missing sanity check for block size during block recovery
[
https://issues.apache.org/jira/browse/HDFS-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981644#comment-14981644
]
Hadoop QA commented on HDFS-9236:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 5s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 48s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 52m 10s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 14s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 19s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 120m 33s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_79 Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-10-29 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12769622/HDFS-9236.004.patch |
| JIRA Issue | HDFS-9236 |
| Optional Tests | asflicense javac javadoc mvninstall unit findbugs
checkstyle compile |
| uname | Linux 7bcaa73db0fe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-c3a2069/precommit/personality/hadoop.sh
|
| git revision | trunk / c293c58 |
| Default Java | 1.7.0_79 |
|
[jira] [Commented] (HDFS-4937) ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom()
[
https://issues.apache.org/jira/browse/HDFS-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981649#comment-14981649
]
Hadoop QA commented on HDFS-4937:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 6s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 21s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 4s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
0s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
8s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 13s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 45s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-10-30 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12769642/HDFS-4937.v2.patch |
| JIRA Issue | HDFS-4937 |
| Optional Tests | asflicense shellcheck javac javadoc mvninstall unit
findbugs checkstyle compile |
| uname | Linux 933890a322fa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
|
| git revision | trunk / e5b1733 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_60
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 |
| shellcheck | v0.4.1 |
| JDK v1.7.0_79 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13285/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDFS-Build/13285/artifact/patchprocess/patch-asflicense-problems.txt
|
|
[jira] [Updated] (HDFS-9303) Balancer slowly with too many small file blocks
[ https://issues.apache.org/jira/browse/HDFS-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Yiqun updated HDFS-9303: Resolution: Duplicate Status: Resolved (was: Patch Available) > Balancer slowly with too many small file blocks > --- > > Key: HDFS-9303 > URL: https://issues.apache.org/jira/browse/HDFS-9303 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9303.001.patch, HDFS-9303.002.patch > > > In the recent hadoop release versions I found that balance operation is > always so slowly, even though I upgrade the version.When I analyse balancer > log, I found that in every balance iteration,it use only 4 to 5 minutes, and > is a short time.And The most important is that the most of being moving > blocks is small blocks ,and the size is smaller than 1M.And this is a main > reason of the low effective of balance operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-9343) Empty caller context considered invalid
[ https://issues.apache.org/jira/browse/HDFS-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reassigned HDFS-9343: --- Assignee: Mingliang Liu > Empty caller context considered invalid > --- > > Key: HDFS-9343 > URL: https://issues.apache.org/jira/browse/HDFS-9343 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Mingliang Liu >Assignee: Mingliang Liu > > The caller context with empty context string is considered invalid, and it > should not appear in the audit log. > Meanwhile, too long signature will not be written to audit log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9343) Empty caller context considered invalid
Mingliang Liu created HDFS-9343: --- Summary: Empty caller context considered invalid Key: HDFS-9343 URL: https://issues.apache.org/jira/browse/HDFS-9343 Project: Hadoop HDFS Issue Type: Task Reporter: Mingliang Liu The caller context with empty context string is considered invalid, and it should not appear in the audit log. Meanwhile, too long signature will not be written to audit log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9343) Empty caller context considered invalid
[ https://issues.apache.org/jira/browse/HDFS-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HDFS-9343: Status: Patch Available (was: Open) > Empty caller context considered invalid > --- > > Key: HDFS-9343 > URL: https://issues.apache.org/jira/browse/HDFS-9343 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Mingliang Liu >Assignee: Mingliang Liu > Attachments: HDFS-9343.000.patch > > > The caller context with empty context string is considered invalid, and it > should not appear in the audit log. > Meanwhile, too long signature will not be written to audit log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9343) Empty caller context considered invalid
[ https://issues.apache.org/jira/browse/HDFS-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HDFS-9343: Attachment: HDFS-9343.000.patch > Empty caller context considered invalid > --- > > Key: HDFS-9343 > URL: https://issues.apache.org/jira/browse/HDFS-9343 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Mingliang Liu >Assignee: Mingliang Liu > Attachments: HDFS-9343.000.patch > > > The caller context with empty context string is considered invalid, and it > should not appear in the audit log. > Meanwhile, too long signature will not be written to audit log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9343) Empty caller context considered invalid
[
https://issues.apache.org/jira/browse/HDFS-9343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981668#comment-14981668
]
Hitesh Shah commented on HDFS-9343:
---
Comments:
isValid() - is this meant to be a public API? Should this be renamed to
isContextValid() as it is only checking the context part?
{code}
if (callerContext.getSignature() != null &&
callerContext.getSignature().length <= callerSignatureMaxLen) {
{code}
- Not checking for zero-length string?
- should signature validity check be moved into a function too?
> Empty caller context considered invalid
> ---
>
> Key: HDFS-9343
> URL: https://issues.apache.org/jira/browse/HDFS-9343
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-9343.000.patch
>
>
> The caller context with empty context string is considered invalid, and it
> should not appear in the audit log.
> Meanwhile, too long signature will not be written to audit log.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
