[ https://issues.apache.org/jira/browse/HADOOP-18545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-18545: ------------------------------------ Summary: hadoop-azure integration tests are failing on the trunk branch against non-HNS stores (was: hadoop-azure integration tests are failing on the trunk branch) > hadoop-azure integration tests are failing on the trunk branch against > non-HNS stores > ------------------------------------------------------------------------------------- > > Key: HADOOP-18545 > URL: https://issues.apache.org/jira/browse/HADOOP-18545 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, test > Affects Versions: 3.4.0 > Reporter: Carl > Priority: Major > > I am getting errors when running the hadoop-azure integration test from the > trunk branch (commit 0ef572abed624011ad9a9e69d725774d6d00b75b). > This is running on an Azure instance, with HNS disabled and using a Shared > Key (NonHNS-SharedKey). > Here are the logs when running those tests: > > {code:java} > [INFO] ------------------------------------------------------- > [INFO] T E S T S > [INFO] ------------------------------------------------------- > [INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsHttpOperation > [INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsClient > [INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsInputStream > [INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAzureADAuthenticator > [INFO] Running > org.apache.hadoop.fs.azurebfs.services.TestShellDecryptionKeyProvider > [INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsOutputStream > [INFO] Running org.apache.hadoop.fs.azurebfs.TestAccountConfiguration > [INFO] Running > org.apache.hadoop.fs.azurebfs.services.TestAbfsClientThrottlingAnalyzer > [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.976 > s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsHttpOperation > [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.174 > s - in org.apache.hadoop.fs.azurebfs.services.TestShellDecryptionKeyProvider > [INFO] Running org.apache.hadoop.fs.azurebfs.services.TestAbfsPerfTracker > [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.921 > s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.TestAccountConfiguration > [ERROR] > testConfigPropNotFound(org.apache.hadoop.fs.azurebfs.TestAccountConfiguration) > Time elapsed: 0.102 s <<< FAILURE! > java.lang.AssertionError: Expected a > org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException > to be thrown, but got the result: : > "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider" > at > org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499) > at > org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384) > at > org.apache.hadoop.fs.azurebfs.TestAccountConfiguration.testMissingConfigKey(TestAccountConfiguration.java:399) > at > org.apache.hadoop.fs.azurebfs.TestAccountConfiguration.testConfigPropNotFound(TestAccountConfiguration.java:386) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at org.junit.runners.ParentRunner.run(ParentRunner.java:413) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) > at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) > [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.396 > s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsClient > [INFO] Running > org.apache.hadoop.fs.azurebfs.services.TestTextFileBasedIdentityHandler > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.043 > s - in org.apache.hadoop.fs.azurebfs.services.TestTextFileBasedIdentityHandler > [INFO] Running org.apache.hadoop.fs.azurebfs.services.TestQueryParams > [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.2 s > - in org.apache.hadoop.fs.azurebfs.services.TestQueryParams > [INFO] Running > org.apache.hadoop.fs.azurebfs.services.TestAbfsRenameRetryRecovery > [INFO] Running > org.apache.hadoop.fs.azurebfs.services.TestExponentialRetryPolicy > [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.088 > s - in org.apache.hadoop.fs.azurebfs.services.TestAzureADAuthenticator > [INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsInputStreamStatistics > [INFO] Running org.apache.hadoop.fs.azurebfs.utils.TestCachedSASToken > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.651 > s - in org.apache.hadoop.fs.azurebfs.utils.TestCachedSASToken > [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.133 > s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsPerfTracker > [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.278 > s - in org.apache.hadoop.fs.azurebfs.services.TestExponentialRetryPolicy > [INFO] Running org.apache.hadoop.fs.azurebfs.utils.TestUriUtils > [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.713 > s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsOutputStream > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.861 > s - in org.apache.hadoop.fs.azurebfs.TestAbfsInputStreamStatistics > [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.668 > s - in org.apache.hadoop.fs.azurebfs.utils.TestUriUtils > [INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsOutputStreamStatistics > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.598 > s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsRenameRetryRecovery > [INFO] Running > org.apache.hadoop.fs.azurebfs.diagnostics.TestConfigurationValidators > [INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsNetworkStatistics > [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.322 > s - in org.apache.hadoop.fs.azurebfs.diagnostics.TestConfigurationValidators > [INFO] Running org.apache.hadoop.fs.azurebfs.TestTracingContext > [INFO] Running > org.apache.hadoop.fs.azurebfs.TestAbfsConfigurationFieldsValidation > [INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsErrorTranslation > [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.61 s > - in org.apache.hadoop.fs.azurebfs.TestAbfsConfigurationFieldsValidation > [INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsStatistics > [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.534 > s - in org.apache.hadoop.fs.azurebfs.TestAbfsErrorTranslation > [INFO] Running > org.apache.hadoop.fs.azurebfs.extensions.TestCustomOauthTokenProvider > [INFO] Running org.apache.hadoop.fs.azurebfs.extensions.TestDTManagerLifecycle > [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.441 > s - in org.apache.hadoop.fs.azurebfs.TestAbfsOutputStreamStatistics > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.849 > s - in org.apache.hadoop.fs.azurebfs.extensions.TestCustomOauthTokenProvider > [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.786 > s - in org.apache.hadoop.fs.azurebfs.TestAbfsNetworkStatistics > [WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: > 6.956 s - in org.apache.hadoop.fs.azurebfs.TestTracingContext > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.59 s > - in org.apache.hadoop.fs.azurebfs.extensions.TestDTManagerLifecycle > [INFO] Running org.apache.hadoop.fs.azurebfs.TestAbfsCrc64 > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.061 > s - in org.apache.hadoop.fs.azurebfs.TestAbfsCrc64 > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.686 > s - in org.apache.hadoop.fs.azurebfs.TestAbfsStatistics > [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.877 > s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsClientThrottlingAnalyzer > [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 55.459 s - in org.apache.hadoop.fs.azurebfs.services.TestAbfsInputStream > [INFO] > [INFO] Results: > [INFO] > [ERROR] Failures: > [ERROR] > TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 > Expected a > org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException > to be thrown, but got the result: : > "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider" > [INFO] > [ERROR] Tests run: 107, Failures: 1, Errors: 0, Skipped: 2 > [INFO] > [ERROR] There are test failures. > Please refer to /hadoop/hadoop-tools/hadoop-azure/target/surefire-reports for > the individual test results. > Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump > and [date].dumpstream. > [INFO] > [INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-azure --- > [INFO] > [INFO] --- maven-jar-plugin:2.5:test-jar (default) @ hadoop-azure --- > [INFO] Building jar: > /hadoop/hadoop-tools/hadoop-azure/target/hadoop-azure-3.4.0-SNAPSHOT-tests.jar > [INFO] > [INFO] --- maven-site-plugin:3.11.0:attach-descriptor (attach-descriptor) @ > hadoop-azure --- > [INFO] Skipping because packaging 'jar' is not pom. > [INFO] > [INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test > (integration-test-abfs-parallel-classesAndMethods) @ hadoop-azure --- > Downloading from central: > https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit47/3.0.0-M1/surefire-junit47-3.0.0-M1.jar > Downloaded from central: > https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit47/3.0.0-M1/surefire-junit47-3.0.0-M1.jar > (93 kB at 2.4 MB/s) > Downloading from central: > https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit47/3.0.0-M1/surefire-junit47-3.0.0-M1.pom > Downloaded from central: > https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit47/3.0.0-M1/surefire-junit47-3.0.0-M1.pom > (7.3 kB at 292 kB/s) > Downloading from central: > https://repo.maven.apache.org/maven2/org/apache/maven/surefire/common-junit48/3.0.0-M1/common-junit48-3.0.0-M1.pom > Downloaded from central: > https://repo.maven.apache.org/maven2/org/apache/maven/surefire/common-junit48/3.0.0-M1/common-junit48-3.0.0-M1.pom > (2.4 kB at 70 kB/s) > Downloading from central: > https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-grouper/3.0.0-M1/surefire-grouper-3.0.0-M1.pom > Downloaded from central: > https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-grouper/3.0.0-M1/surefire-grouper-3.0.0-M1.pom > (2.6 kB at 146 kB/s) > Downloading from central: > https://repo.maven.apache.org/maven2/org/apache/maven/surefire/common-junit48/3.0.0-M1/common-junit48-3.0.0-M1.jar > Downloading from central: > https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-grouper/3.0.0-M1/surefire-grouper-3.0.0-M1.jar > Downloaded from central: > https://repo.maven.apache.org/maven2/org/apache/maven/surefire/common-junit48/3.0.0-M1/common-junit48-3.0.0-M1.jar > (22 kB at 1.3 MB/s) > Downloaded from central: > https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-grouper/3.0.0-M1/surefire-grouper-3.0.0-M1.jar > (40 kB at 1.8 MB/s) > [INFO] > [INFO] ------------------------------------------------------- > [INFO] T E S T S > [INFO] ------------------------------------------------------- > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsRestOperationException > [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.294 > s - in org.apache.hadoop.fs.azurebfs.ITestAbfsRestOperationException > [INFO] Running > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRenameUnicode > [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.288 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRenameUnicode > [INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsOutputStream > [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.765 > s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsOutputStream > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCLI > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.222 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCLI > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics > [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.588 > s - in org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics > [INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsPositionedRead > [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.635 > s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsPositionedRead > [INFO] Running > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.634 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat > [INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsUnbuffer > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.496 > s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsUnbuffer > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCopy > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.503 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCopy > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAppend > [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.634 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAppend > [INFO] Running > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAuthorization > [WARNING] Tests run: 29, Failures: 0, Errors: 0, Skipped: 29, Time elapsed: > 2.638 s - in > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAuthorization > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestOauthOverAbfsScheme > [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: > 0.277 s - in org.apache.hadoop.fs.azurebfs.ITestOauthOverAbfsScheme > [INFO] Running > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemInitAndCreate > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.317 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemInitAndCreate > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsDurationTrackers > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.97 s > - in org.apache.hadoop.fs.azurebfs.ITestAbfsDurationTrackers > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelete > [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.973 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelete > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsClient > [WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: > 19.652 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsClient > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFilesystemAcl > [WARNING] Tests run: 66, Failures: 0, Errors: 0, Skipped: 56, Time elapsed: > 11.749 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFilesystemAcl > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestFileSystemProperties > [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.013 > s - in org.apache.hadoop.fs.azurebfs.ITestFileSystemProperties > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemOauth > [WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: > 0.099 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemOauth > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemMkDir > [WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: > 0.549 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemMkDir > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsOutputStreamStatistics > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.074 > s - in org.apache.hadoop.fs.azurebfs.ITestAbfsOutputStreamStatistics > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsIdentityTransformer > [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.745 > s - in org.apache.hadoop.fs.azurebfs.ITestAbfsIdentityTransformer > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFlush > [INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 23.404 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFlush > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2E > [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.549 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2E > [INFO] Running > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemStoreListStatusWithRange > [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.767 > s - in > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemStoreListStatusWithRange > [INFO] Running > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess > [ERROR] Tests run: 12, Failures: 1, Errors: 0, Skipped: 10, Time elapsed: > 1.521 s <<< FAILURE! - in > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess > [ERROR] > testCheckAccessForAccountWithoutNS(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess) > Time elapsed: 1.15 s <<< FAILURE! > java.lang.AssertionError: Expecting > org.apache.hadoop.security.AccessControlException with text "This request is > not authorized to perform this operation using this permission.", 403 but got > : "void" > at > org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499) > at > org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:529) > at > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS(ITestAzureBlobFileSystemCheckAccess.java:181) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at java.base/java.lang.Thread.run(Thread.java:829) > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsInputStreamStatistics > [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.827 > s - in org.apache.hadoop.fs.azurebfs.ITestAbfsInputStreamStatistics > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsStatistics > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.649 > s - in org.apache.hadoop.fs.azurebfs.ITestAbfsStatistics > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCreate > [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.393 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCreate > [INFO] Running > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemMainOperation > [INFO] Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.373 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemMainOperation > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFinalize > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.362 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFinalize > [INFO] Running > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRandomRead > [WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: > 3.196 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRandomRead > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename > [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.622 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsHugeFiles > [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.34 > s - in org.apache.hadoop.fs.azurebfs.ITestAbfsHugeFiles > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility > [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.802 > s - in org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestClientUrlScheme > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.604 > s - in org.apache.hadoop.fs.azurebfs.ITestClientUrlScheme > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestFileSystemRegistration > [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.775 > s - in org.apache.hadoop.fs.azurebfs.ITestFileSystemRegistration > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestSharedKeyAuth > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.186 > s - in org.apache.hadoop.fs.azurebfs.ITestSharedKeyAuth > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestFileSystemInitialization > [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.474 > s - in org.apache.hadoop.fs.azurebfs.ITestFileSystemInitialization > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestCustomerProvidedKey > [WARNING] Tests run: 28, Failures: 0, Errors: 0, Skipped: 28, Time elapsed: > 0.882 s - in org.apache.hadoop.fs.azurebfs.ITestCustomerProvidedKey > [INFO] Running > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAttributes > [WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: > 0.402 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAttributes > [INFO] Running > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelegationSAS > [WARNING] Tests run: 14, Failures: 0, Errors: 0, Skipped: 14, Time elapsed: > 0.41 s - in > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelegationSAS > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsMsiTokenProvider > [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: > 0.298 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsMsiTokenProvider > [INFO] Running > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFileStatus > [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.921 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFileStatus > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestGetNameSpaceEnabled > [WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: > 6.37 s - in org.apache.hadoop.fs.azurebfs.ITestGetNameSpaceEnabled > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsListStatusRemoteIterator > [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.053 > s - in org.apache.hadoop.fs.azurebfs.ITestAbfsListStatusRemoteIterator > [INFO] Running > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemPermission > [WARNING] Tests run: 128, Failures: 0, Errors: 0, Skipped: 128, Time elapsed: > 9.17 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemPermission > [INFO] Running > org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStreamSmallFileReads > [INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 48.622 s - in > org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStreamSmallFileReads > [INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStream > [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.918 > s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStream > [INFO] Running > org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStreamReadFooter > [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 58.717 s - in > org.apache.hadoop.fs.azurebfs.services.ITestAbfsInputStreamReadFooter > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease > [INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 75.544 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease > [INFO] > [INFO] Results: > [INFO] > [ERROR] Failures: > [ERROR] > ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS:181 > Expecting org.apache.hadoop.security.AccessControlException with text "This > request is not authorized to perform this operation using this permission.", > 403 but got : "void" > [INFO] > [ERROR] Tests run: 567, Failures: 1, Errors: 0, Skipped: 277 > [INFO] > [INFO] > [INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test > (integration-test-abfs-parallel-classes) @ hadoop-azure --- > [INFO] > [INFO] ------------------------------------------------------- > [INFO] T E S T S > [INFO] ------------------------------------------------------- > [INFO] Running > org.apache.hadoop.fs.azurebfs.commit.ITestAbfsLoadManifestsStage > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestSmallWriteOptimization > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek > [INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestReadBufferManager > [INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCleanupStage > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale > [INFO] Running > org.apache.hadoop.fs.azurebfs.commit.ITestAbfsRenameStageFailure > [INFO] Running > org.apache.hadoop.fs.azurebfs.commit.ITestAbfsJobThroughManifestCommitter > [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.918 > s - in org.apache.hadoop.fs.azurebfs.services.ITestReadBufferManager > [INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCommitTaskStage > [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.879 > s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsRenameStageFailure > [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.657 > s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCleanupStage > [INFO] Running > org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTaskManifestFileIO > [INFO] Running > org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCreateOutputDirectoriesStage > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.991 > s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsLoadManifestsStage > [INFO] Running > org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol > [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.569 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale > [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.159 > s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCommitTaskStage > [INFO] Running org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort > [INFO] Running > org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestStoreOperations > [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.197 > s - in org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTaskManifestFileIO > [ERROR] Tests run: 19, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > 32.672 s <<< FAILURE! - in > org.apache.hadoop.fs.azurebfs.commit.ITestAbfsJobThroughManifestCommitter > [ERROR] > test_0420_validateJob(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsJobThroughManifestCommitter) > Time elapsed: 0.74 s <<< ERROR! > org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: > > `abfs://****@****.dfs.core.windows.net/fork-0002/test/TestJobThroughManifestCommitter/out%20put/dir-03-00/dir-02-00/dir-01-00/part-00-0033': > Expected the file renamed from > abfs://****@****.dfs.core.windows.net/fork-0002/test/TestJobThroughManifestCommitter/out > > put/_temporary/202211291549220355_0002/01/tasks/attempt_202211291549220355_0002_m_000000_1/dir-03-00/dir-02-00/dir-01-00/part-00-0033 > with etag 0x8DAD22153D4F458 and length 2 but found a file with etag > 0x8DAD22159994F7E and length 2 > at > org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166) > at > org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 33.371 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek > [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRename > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAzureBlobFileSystemBasics > [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.86 > s - in > org.apache.hadoop.fs.azurebfs.commit.ITestAbfsCreateOutputDirectoriesStage > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRootDirectory > [WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: > 8.559 s - in > org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestStoreOperations > [WARNING] Tests run: 48, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: > 41.978 s - in org.apache.hadoop.fs.azurebfs.ITestSmallWriteOptimization > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractOpen > [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.96 s > - in org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractMkdir > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSetTimes > [WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 9, Time elapsed: > 9.624 s - in > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRootDirectory > [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 14.207 s - in > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRename > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractEtag > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsContractUnbuffer > [ERROR] Tests run: 17, Failures: 0, Errors: 7, Skipped: 0, Time elapsed: > 25.661 s <<< FAILURE! - in > org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol > [ERROR] > testCommitterWithDuplicatedCommit(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) > Time elapsed: 10.143 s <<< ERROR! > org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: > > `abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testCommitterWithDuplicatedCommit/part-m-00000': > Expected the file renamed from > abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testCommitterWithDuplicatedCommit/_temporary/job_202211291549500881_0004/01/tasks/attempt_202211291549500881_0004_m_000000_0/part-m-00000 > with etag 0x8DAD2215D8BA025 and length 40 but found a file with etag > 0x8DAD2215F25DA02 and length 40 > at > org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166) > at > org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > [ERROR] > testMapFileOutputCommitter(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) > Time elapsed: 1.857 s <<< ERROR! > org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: > > `abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testMapFileOutputCommitter/part-m-00000/data': > Expected the file renamed from > abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testMapFileOutputCommitter/_temporary/job_202211291550000055_0004/01/tasks/attempt_202211291550000055_0004_m_000000_0/part-m-00000/data > with etag 0x8DAD2215FF58EA2 and length 296 but found a file with etag > 0x8DAD221603B8E0D and length 296 > at > org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166) > at > org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > [ERROR] > testCommitLifecycle(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) > Time elapsed: 1.462 s <<< ERROR! > org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: > > `abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testCommitLifecycle/part-m-00000': > Expected the file renamed from > abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testCommitLifecycle/_temporary/job_202211291550020019_0004/01/tasks/attempt_202211291550020019_0004_m_000000_0/part-m-00000 > with etag 0x8DAD22160F41493 and length 40 but found a file with etag > 0x8DAD22161466E35 and length 40 > at > org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166) > at > org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > [ERROR] > testConcurrentCommitTaskWithSubDir(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) > Time elapsed: 1.262 s <<< ERROR! > org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: > > `abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testConcurrentCommitTaskWithSubDir/SUB_DIR/part-m-00000': > Expected the file renamed from > abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testConcurrentCommitTaskWithSubDir/_temporary/job_202211291550030237_0004/00/tasks/attempt_202211291550030237_0004_m_000000_1/SUB_DIR/part-m-00000 > with etag 0x8DAD22161DB6C66 and length 40 but found a file with etag > 0x8DAD22162100941 and length 40 > at > org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166) > at > org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > [ERROR] > testParallelJobsToAdjacentPaths(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) > Time elapsed: 1.401 s <<< ERROR! > org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: > > `abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testParallelJobsToAdjacentPaths/part-m-00000': > Expected the file renamed from > abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testParallelJobsToAdjacentPaths/_temporary/job_202211291550080667_0004/01/tasks/attempt_202211291550080667_0004_m_000000_0/part-m-00000 > with etag 0x8DAD221646E60AB and length 40 but found a file with etag > 0x8DAD22164B4AE25 and length 40 > at > org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166) > at > org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > [ERROR] > testOutputFormatIntegration(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) > Time elapsed: 0.925 s <<< ERROR! > org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: > > `abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testOutputFormatIntegration/part-m-00000': > Expected the file renamed from > abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testOutputFormatIntegration/_temporary/job_202211291550110055_0004/01/tasks/attempt_202211291550110055_0004_m_000000_0/part-m-00000 > with etag 0x8DAD221662398A8 and length 4 but found a file with etag > 0x8DAD221664D1399 and length 4 > at > org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166) > at > org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > [ERROR] > testTwoTaskAttemptsCommit(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsManifestCommitProtocol) > Time elapsed: 1.291 s <<< ERROR! > org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.OutputValidationException: > > `abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testTwoTaskAttemptsCommit/attempt2-m-00000': > Expected the file renamed from > abfs://****@****.dfs.core.windows.net/fork-0004/test/ITestAbfsManifestCommitProtocol-testTwoTaskAttemptsCommit/_temporary/job_202211291550130427_0004/01/tasks/attempt_202211291550130427_0004_m_000000_1/attempt2-m-00000 > with etag 0x8DAD221676CB12D and length 40 but found a file with etag > 0x8DAD22167ACBE08 and length 40 > at > org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.ValidateRenamedFilesStage.validateOneFile(ValidateRenamedFilesStage.java:166) > at > org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: > 7.58 s - in > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSetTimes > [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.177 > s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractMkdir > [INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 11.624 s - in > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractOpen > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractAppend > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractConcat > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDelete > [INFO] Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.35 > s - in org.apache.hadoop.fs.azurebfs.contract.ITestAzureBlobFileSystemBasics > [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.322 > s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsContractUnbuffer > [WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: > 11.125 s - in > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractEtag > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractCreate > [WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: > 6.888 s - in > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractConcat > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractGetFileStatus > [INFO] Running > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSeek > [WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: > 9.429 s - in > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractAppend > [INFO] Running > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus > [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.189 > s - in > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDelete > [INFO] Running > org.apache.hadoop.fs.azurebfs.extensions.ITestAbfsDelegationTokens > [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 13.352 s - in > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractCreate > [INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 13.206 s - in > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSeek > [INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 15.014 s - in > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractGetFileStatus > [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.823 > s - in org.apache.hadoop.fs.azurebfs.extensions.ITestAbfsDelegationTokens > [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.969 > s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus > [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 49.734 s - in > org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp > [ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: > 100.527 s <<< FAILURE! - in > org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort > [ERROR] > test_110_teragen(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort) > Time elapsed: 25.881 s <<< FAILURE! > java.lang.AssertionError: teragen(1000, > abfs://****@****.dfs.core.windows.net/ITestAbfsTerasort/sortin) failed > expected:<0> but was:<1> > at org.junit.Assert.fail(Assert.java:89) > at org.junit.Assert.failNotEquals(Assert.java:835) > at org.junit.Assert.assertEquals(Assert.java:647) > at > org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort.executeStage(ITestAbfsTerasort.java:211) > at > org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort.test_110_teragen(ITestAbfsTerasort.java:244) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at java.base/java.lang.Thread.run(Thread.java:829) > [INFO] > [INFO] Results: > [INFO] > [ERROR] Failures: > [ERROR] > ITestAbfsTerasort.test_110_teragen:244->executeStage:211->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 > teragen(1000, > abfs://****@****.dfs.core.windows.net/ITestAbfsTerasort/sortin) failed > expected:<0> but was:<1> > [ERROR] Errors: > [ERROR] ITestAbfsJobThroughManifestCommitter.test_0420_validateJob ? > OutputValidation ... > [ERROR] ITestAbfsManifestCommitProtocol.testCommitLifecycle ? > OutputValidation `abfs:/... > [ERROR] ITestAbfsManifestCommitProtocol.testCommitterWithDuplicatedCommit ? > OutputValidation > [ERROR] ITestAbfsManifestCommitProtocol.testConcurrentCommitTaskWithSubDir > ? OutputValidation > [ERROR] ITestAbfsManifestCommitProtocol.testMapFileOutputCommitter ? > OutputValidation ... > [ERROR] ITestAbfsManifestCommitProtocol.testOutputFormatIntegration ? > OutputValidation > [ERROR] ITestAbfsManifestCommitProtocol.testParallelJobsToAdjacentPaths ? > OutputValidation > [ERROR] ITestAbfsManifestCommitProtocol.testTwoTaskAttemptsCommit ? > OutputValidation `... > [INFO] > [ERROR] Tests run: 335, Failures: 1, Errors: 8, Skipped: 46 {code} > > And here are the commands I run on a new instance to get there: > > {code:java} > cd /hadoop/hadoop-maven-plugins > mvn install > cd /hadoop/hadoop-tools/hadoop-azure > cp ./src/test/resources/azure-auth-keys.xml.template > ./src/test/resources/azure-auth-keys.xml > az login --identity > ACCOUNT_NAME=**** > CONTAINER_NAME=**** > RESOURCE_GROUP=**** > ACCOUNT_ACCESS_KEY=$(az storage account keys list \ > --resource-group "$RESOURCE_GROUP" \ > --account-name "$ACCOUNT_NAME" \ > --output tsv \ > --query '[].{value:value} | [0]') > cat <<EOT > src/test/resources/azure-auth-keys.xml > <?xml version="1.0" encoding="UTF-8"?> > <configuration> > <!-- Account names without domain --> > <property> > <name>fs.azure.hnsTestAccountName</name> > <value>$ACCOUNT_NAME</value> > </property> > <property> > <name>fs.azure.nonHnsTestAccountName</name> > <value>$ACCOUNT_NAME</value> > </property> > </configuration> > EOT > cp ./src/test/resources/accountSettings/accountName_settings.xml.template > ./src/test/resources/accountSettings/${ACCOUNT_NAME}_settings.xml > sed -i -e "s/ACCOUNTNAME/$ACCOUNT_NAME/g" > ./src/test/resources/accountSettings/${ACCOUNT_NAME}_settings.xml > sed -i -e "s/CONTAINER_NAME/$CONTAINER_NAME/g" > ./src/test/resources/accountSettings/${ACCOUNT_NAME}_settings.xml > sed -i -e "s/IS_NAMESPACE_ENABLED/false/g" > ./src/test/resources/accountSettings/${ACCOUNT_NAME}_settings.xml > sed -i -e "s#ACCOUNT_KEY#$ACCOUNT_ACCESS_KEY#g" > ./src/test/resources/accountSettings/${ACCOUNT_NAME}_settings.xml > dev-support/testrun-scripts/runtests.sh{code} > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org