[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949782#comment-16949782 ]
Jeetesh Mangwani commented on HADOOP-16612: ------------------------------------------- I have fixed the checkstyle issues. Please take a look. When I run tests on my machine, I see 4 failing tests on my 'trunk' branch. These tests fail on my feature branch too. 1. ITestGetNameSpaceEnabled.testNonXNSAccount: fails because the HTTP response status is not 400, but is 404 2. ITestAzureBlobFileSystemCLI.testMkdirRootNonExistentContainer: fails in the setup phase because the HTTP response status is not 400, but is 404 3. ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek: times out, probably because this is a scale test and my machine is slow 4. ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads: times out, probably because there are lot of heavy writes and my machine is slow Here are the details: trunk, non-xns [INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Failures: [ERROR] ITestGetNameSpaceEnabled.testNonXNSAccount:59->Assert.assertFalse:64->Assert.assertTrue:41->Assert.fail:88 Expecting getIsNamespaceEnabled() return false [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemCLI>AbstractAbfsIntegrationTest.setup:137 » AbfsRestOperation [INFO] [ERROR] Tests run: 395, Failures: 1, Errors: 1, Skipped: 21 [ERROR] Errors: [ERROR] ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:60->testReadWriteAndSeek:75 » TestTimedOut [ERROR] ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 » TestTimedOut [INFO] [ERROR] Tests run: 192, Failures: 0, Errors: 2, Skipped: 24 =================================== feature branch, non-xns [INFO] Tests run: 53, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Failures: [ERROR] ITestGetNameSpaceEnabled.testNonXNSAccount:59->Assert.assertFalse:64->Assert.assertTrue:41->Assert.fail:88 Expecting getIsNamespaceEnabled() return false [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemCLI>AbstractAbfsIntegrationTest.setup:137 » AbfsRestOperation [INFO] [ERROR] Tests run: 395, Failures: 1, Errors: 1, Skipped: 21 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:60->testReadWriteAndSeek:75 » TestTimedOut [ERROR] ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 » TestTimedOut [INFO] [ERROR] Tests run: 192, Failures: 0, Errors: 2, Skipped: 24 ================================ trunk, xns [INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemCLI>AbstractAbfsIntegrationTest.setup:137 » AbfsRestOperation [INFO] [ERROR] Tests run: 395, Failures: 0, Errors: 1, Skipped: 21 [ERROR] Errors: [ERROR] ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:60->testReadWriteAndSeek:75 » TestTimedOut [ERROR] ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 » TestTimedOut [INFO] [ERROR] Tests run: 192, Failures: 0, Errors: 2, Skipped: 24 ================================= feature branch, xns [INFO] Tests run: 53, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemCLI>AbstractAbfsIntegrationTest.setup:137 » AbfsRestOperation [INFO] [ERROR] Tests run: 395, Failures: 0, Errors: 1, Skipped: 21 [ERROR] Errors: [ERROR] ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:60->testReadWriteAndSeek:75 » TestTimedOut [ERROR] ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 » TestTimedOut [INFO] [ERROR] Tests run: 192, Failures: 0, Errors: 2, Skipped: 24 Error stack traces: [ERROR] testMkdirRootNonExistentContainer(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCLI) Time elapsed: 18.27 s <<< ERROR! Operation failed: "The specified filesystem does not exist.", 404, HEAD, https://abfstest02.dfs.core.windows.net/abfs-testcontainer-aa478873-647e-455a-9a71-4cb6da30a088//?upn=false&action=getAccessControl&timeout=90 at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:143) at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:523) at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:506) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:224) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getIsNamespaceEnabled(AzureBlobFileSystem.java:1108) at org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.setup(AbstractAbfsIntegrationTest.java:137) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) [ERROR] testNonXNSAccount(org.apache.hadoop.fs.azurebfs.ITestGetNameSpaceEnabled) Time elapsed: 0.808 s <<< FAILURE! java.lang.AssertionError: Expecting getIsNamespaceEnabled() return false at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertFalse(Assert.java:64) at org.apache.hadoop.fs.azurebfs.ITestGetNameSpaceEnabled.testNonXNSAccount(ITestGetNameSpaceEnabled.java:59) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) [ERROR] testReadAndWriteWithDifferentBufferSizesAndSeek[Size=104,857,600](org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek) Time elapsed: 1,800.017 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 1800000 milliseconds at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429) at java.util.concurrent.FutureTask.get(FutureTask.java:191) at org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.flushWrittenBytesToService(AbfsOutputStream.java:317) at org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.flushInternal(AbfsOutputStream.java:265) at org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.close(AbfsOutputStream.java:248) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) at org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek.testReadWriteAndSeek(ITestAbfsReadWriteAndSeek.java:75) at org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek(ITestAbfsReadWriteAndSeek.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1,970.475 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale [ERROR] testWriteHeavyBytesToFileAcrossThreads(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale) Time elapsed: 1,800.018 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 1800000 milliseconds at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429) at java.util.concurrent.FutureTask.get(FutureTask.java:191) at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads(ITestAzureBlobFileSystemE2EScale.java:77) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) > Track Azure Blob File System client-perceived latency > ----------------------------------------------------- > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client > Reporter: Jeetesh Mangwani > Assignee: Jeetesh Mangwani > Priority: Major > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring > latencies in the Hadoop ABFS driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org