[ https://issues.apache.org/jira/browse/HADOOP-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509107#comment-16509107 ]
Da Zhou commented on HADOOP-15407: ---------------------------------- Submitting HADOOP-15407-HADOOP-15407.008.patch, all ABFS tests passed against my storage account in west US. Updates in the patch: - Resolved findbugs violations - Resolved checkstyle violations - Added missing java docs - Updated AzureBlobFileSystemException as a subclass of IOE and updated exception check - Reinstate wasb contract tests to parallel runs, enable the parallel runs for ABFS contract tests - Updated names of Service injection interface and impl with Azure specific names - Replaced loggingService with SL4J {noformat} mvn -T 1C -Dparallel-tests -DtestsThreadCount=8 clean verify [INFO] --- maven-antrun-plugin:1.7:run (create-parallel-tests-dirs) @ hadoop-azure --- [INFO] Executing tasks main: [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test-dir/1 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test-dir/2 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test-dir/3 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test-dir/4 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test-dir/5 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test-dir/6 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test-dir/7 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test-dir/8 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test/1 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test/2 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test/3 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test/4 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test/5 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test/6 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test/7 [mkdir] Created dir: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/test/8 [INFO] Executed tasks [INFO] [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ hadoop-azure --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider [INFO] Running org.apache.hadoop.fs.azure.TestBlobMetadata [INFO] Running org.apache.hadoop.fs.azure.TestWasbFsck [INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked [INFO] Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations [INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic [INFO] Running org.apache.hadoop.fs.azure.TestClientThrottlingAnalyzer [INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked [WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 1.098 s - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic [WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 1.934 s - in org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider [INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency [INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked [WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 8.097 s - in org.apache.hadoop.fs.azure.TestWasbFsck [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.808 s - in org.apache.hadoop.fs.azure.TestBlobMetadata [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.033 s - in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations [INFO] Running org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem [INFO] Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater [INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.308 s - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.594 s - in org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater [INFO] Running org.apache.hadoop.fs.azure.TestBlobOperationDescriptor [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.877 s - in org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem [INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck [INFO] Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.432 s - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked [WARNING] Tests run: 59, Failures: 0, Errors: 0, Skipped: 59, Time elapsed: 9.13 s - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization [WARNING] Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 18.249 s - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked [INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockCompaction [WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 3.602 s - in org.apache.hadoop.fs.azure.TestBlobOperationDescriptor [INFO] Running org.apache.hadoop.fs.azurebfs.diagnostics.TestConfigurationValidators [INFO] Running org.apache.hadoop.fs.azurebfs.utils.TestUriUtils [INFO] Running org.apache.hadoop.fs.azurebfs.services.TestConfigurationServiceFieldsValidation [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 s - in org.apache.hadoop.fs.azurebfs.utils.TestUriUtils [WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.604 s - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockCompaction [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.547 s - in org.apache.hadoop.fs.azurebfs.services.TestConfigurationServiceFieldsValidation [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.761 s - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.051 s - in org.apache.hadoop.fs.azurebfs.diagnostics.TestConfigurationValidators [INFO] Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.703 s - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.361 s - in org.apache.hadoop.fs.azure.TestClientThrottlingAnalyzer [INFO] [INFO] Results: [INFO] [WARNING] Tests run: 257, Failures: 0, Errors: 0, Skipped: 76 [INFO] [INFO] [INFO] --- maven-surefire-plugin:2.21.0:test (serialized-test) @ hadoop-azure --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.263 s - in org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage [INFO] [INFO] Results: [INFO] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] [INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-azure --- [INFO] Building jar: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/hadoop-azure-3.2.0-SNAPSHOT.jar [INFO] [INFO] --- maven-jar-plugin:2.5:test-jar (default) @ hadoop-azure --- [INFO] Building jar: /home/zhoda/dev/Projects/apache-hadoop/hadoop/hadoop-tools/hadoop-azure/target/hadoop-azure-3.2.0-SNAPSHOT-tests.jar [INFO] [INFO] --- maven-site-plugin:3.6:attach-descriptor (attach-descriptor) @ hadoop-azure --- [INFO] Skipping because packaging 'jar' is not pom. [INFO] [INFO] --- maven-failsafe-plugin:2.21.0:integration-test (default-integration-test) @ hadoop-azure --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.apache.hadoop.fs.azure.ITestPageBlobInputStream [INFO] Running org.apache.hadoop.fs.azure.ITestOutputStreamSemantics [INFO] Running org.apache.hadoop.fs.azure.ITestWasbUriAndConfiguration [INFO] Running org.apache.hadoop.fs.azure.ITestNativeAzureFSAuthWithBlobSpecificKeys [INFO] Running org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles [INFO] Running org.apache.hadoop.fs.azure.ITestFileSystemOperationExceptionMessage [INFO] Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend [INFO] Running org.apache.hadoop.fs.azure.ITestBlobDataValidation [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 2.934 s - in org.apache.hadoop.fs.azure.ITestFileSystemOperationExceptionMessage [WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 4.144 s - in org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles [WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 4.644 s - in org.apache.hadoop.fs.azure.ITestPageBlobInputStream [WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 4.162 s - in org.apache.hadoop.fs.azure.ITestBlobDataValidation [WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 4.438 s - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend [WARNING] Tests run: 15, Failures: 0, Errors: 0, Skipped: 15, Time elapsed: 5.119 s - in org.apache.hadoop.fs.azure.ITestOutputStreamSemantics [INFO] Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen [INFO] Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek [INFO] Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename [INFO] Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDistCp [INFO] Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete [INFO] Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus [WARNING] Tests run: 59, Failures: 0, Errors: 0, Skipped: 59, Time elapsed: 8.834 s - in org.apache.hadoop.fs.azure.ITestNativeAzureFSAuthWithBlobSpecificKeys [WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 3.283 s - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen [WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 2.725 s - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename [WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 2.84 s - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDistCp [WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 3.006 s - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete [WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 4.381 s - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek [WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 3.474 s - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus [INFO] Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend [INFO] Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate [INFO] Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir [INFO] Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging [INFO] Running org.apache.hadoop.fs.azure.ITestNativeAzureFSAuthorizationCaching [INFO] Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo [INFO] Running org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite [WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 3.264 s - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend [WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 3.797 s - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate [WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 3.465 s - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir [WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 3.766 s - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging [INFO] Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 3.492 s - in org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo [INFO] Running org.apache.hadoop.fs.azure.ITestAzureFileSystemErrorConditions [WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 4.097 s - in org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite [INFO] Running org.apache.hadoop.fs.azure.metrics.ITestAzureFileSystemInstrumentation [INFO] Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator [INFO] Running org.apache.hadoop.fs.azure.ITestFileSystemOperationExceptionHandling [INFO] Running org.apache.hadoop.fs.azure.ITestBlobTypeSpeedDifference [WARNING] Tests run: 19, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 21.663 s - in org.apache.hadoop.fs.azure.ITestWasbUriAndConfiguration [WARNING] Tests run: 60, Failures: 0, Errors: 0, Skipped: 60, Time elapsed: 9.332 s - in org.apache.hadoop.fs.azure.ITestNativeAzureFSAuthorizationCaching [WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 4.571 s - in org.apache.hadoop.fs.azure.metrics.ITestAzureFileSystemInstrumentation [WARNING] Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 6.209 s - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive [INFO] Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAtomicRenameDirList [INFO] Running org.apache.hadoop.fs.azure.ITestContainerChecks [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 3.748 s - in org.apache.hadoop.fs.azure.ITestBlobTypeSpeedDifference [WARNING] Tests run: 16, Failures: 0, Errors: 0, Skipped: 16, Time elapsed: 5.071 s - in org.apache.hadoop.fs.azure.ITestFileSystemOperationExceptionHandling [WARNING] Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 6.254 s - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator [INFO] Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive [INFO] Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemInitAndCreate [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 3.817 s - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAtomicRenameDirList [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2E [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFileStatus [WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 4.185 s - in org.apache.hadoop.fs.azure.ITestContainerChecks [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 3.799 s - in org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode [WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 12.574 s - in org.apache.hadoop.fs.azure.ITestAzureFileSystemErrorConditions [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCopy [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFlush [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemOpen [WARNING] Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 7.222 s - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive [INFO] Running org.apache.hadoop.fs.azurebfs.ITestFileSystemRegistration [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.281 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemInitAndCreate [INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractGetFileStatus [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.226 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFileStatus [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.13 s - in org.apache.hadoop.fs.azurebfs.ITestFileSystemRegistration [INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDelete [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.693 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemOpen [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.993 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCopy [INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractCreate [INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractOpen [INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRename [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.745 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2E [INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAzureBlobFileSystemBasics [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.871 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractOpen [INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSecureDistCp [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.042 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.773 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDelete [INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSeek [INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractMkdir [WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 7.411 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSecureDistCp [INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRootDirectory [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.058 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRename [INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractConcat [INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.601 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractCreate [INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp [WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 6.903 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp [WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 16.737 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractRootDirectory [INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractAppend [INFO] Running org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSetTimes [WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 13.34 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractConcat [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus [INFO] Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.025 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractGetFileStatus [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.52 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractMkdir [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale [WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 12.149 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSetTimes [INFO] Running org.apache.hadoop.fs.azurebfs.ITestFileSystemInitialization [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat [WARNING] Tests run: 14, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 23.135 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractAppend [INFO] Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.626 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAzureBlobFileSystemBasics [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCreate [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelete [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.462 s - in org.apache.hadoop.fs.azurebfs.ITestFileSystemInitialization [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemMkDir [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.766 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat [INFO] Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 58.708 s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractSeek [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAppend [INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestTracingServiceImpl [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.283 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCreate [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.992 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemMkDir [INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsHttpServiceImpl [INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestReadWriteAndSeek [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.389 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAppend [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.655 s - in org.apache.hadoop.fs.azurebfs.services.ITestTracingServiceImpl [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.149 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemDelete [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.06 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus [WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 16.201 s - in org.apache.hadoop.fs.azurebfs.services.ITestAbfsHttpServiceImpl [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 152.018 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFlush [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 122.774 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 156.677 s - in org.apache.hadoop.fs.azurebfs.services.ITestReadWriteAndSeek [INFO] [INFO] Results: [INFO] [WARNING] Tests run: 754, Failures: 0, Errors: 0, Skipped: 471 [INFO] [INFO] [INFO] --- maven-failsafe-plugin:2.21.0:integration-test (sequential-integration-tests) @ hadoop-azure --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive [WARNING] Tests run: 52, Failures: 0, Errors: 0, Skipped: 52, Time elapsed: 0.814 s - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive [INFO] Running org.apache.hadoop.fs.azure.ITestOutOfBandAzureBlobOperationsLive [WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 9, Time elapsed: 0.042 s - in org.apache.hadoop.fs.azure.ITestOutOfBandAzureBlobOperationsLive [INFO] Running org.apache.hadoop.fs.azure.ITestNativeAzureFSPageBlobLive [WARNING] Tests run: 46, Failures: 0, Errors: 0, Skipped: 46, Time elapsed: 0.46 s - in org.apache.hadoop.fs.azure.ITestNativeAzureFSPageBlobLive [INFO] Running org.apache.hadoop.fs.azure.ITestBlockBlobInputStream [WARNING] Tests run: 21, Failures: 0, Errors: 0, Skipped: 21, Time elapsed: 0.473 s - in org.apache.hadoop.fs.azure.ITestBlockBlobInputStream [INFO] Running org.apache.hadoop.fs.azure.ITestFileSystemOperationsWithThreads [WARNING] Tests run: 20, Failures: 0, Errors: 0, Skipped: 20, Time elapsed: 0.194 s - in org.apache.hadoop.fs.azure.ITestFileSystemOperationsWithThreads [INFO] Running org.apache.hadoop.fs.azure.ITestWasbRemoteCallHelper [WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 0.079 s - in org.apache.hadoop.fs.azure.ITestWasbRemoteCallHelper [INFO] Running org.apache.hadoop.fs.azure.ITestFileSystemOperationsExceptionHandlingMultiThreaded [WARNING] Tests run: 12, Failures: 0, Errors: 0, Skipped: 12, Time elapsed: 0.06 s - in org.apache.hadoop.fs.azure.ITestFileSystemOperationsExceptionHandlingMultiThreaded [INFO] Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemConcurrencyLive [WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.008 s - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemConcurrencyLive [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRandomRead [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 81.087 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRandomRead [INFO] Running org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.453 s - in org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility [INFO] [INFO] Results: [INFO] [WARNING] Tests run: 186, Failures: 0, Errors: 0, Skipped: 172 [INFO] [INFO] [INFO] --- animal-sniffer-maven-plugin:1.16:check (signature-check) @ hadoop-azure --- [INFO] Checking unresolved references to org.codehaus.mojo.signature:java18:1.0 [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (depcheck) @ hadoop-azure --- [INFO] [INFO] --- maven-failsafe-plugin:2.21.0:verify (default-integration-test) @ hadoop-azure --- [INFO] [INFO] --- maven-failsafe-plugin:2.21.0:verify (sequential-integration-tests) @ hadoop-azure --- [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ---------------------------------------------------------- {noformat} > Support Windows Azure Storage - Blob file system in Hadoop > ---------------------------------------------------------- > > Key: HADOOP-15407 > URL: https://issues.apache.org/jira/browse/HADOOP-15407 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure > Affects Versions: 3.2.0 > Reporter: Esfandiar Manii > Assignee: Esfandiar Manii > Priority: Major > Attachments: HADOOP-15407-001.patch, HADOOP-15407-002.patch, > HADOOP-15407-003.patch, HADOOP-15407-004.patch, > HADOOP-15407-HADOOP-15407.006.patch, HADOOP-15407-HADOOP-15407.007.patch, > HADOOP-15407-patch-atop-patch-007.patch > > > *{color:#212121}Description{color}* > This JIRA adds a new file system implementation, ABFS, for running Big Data > and Analytics workloads against Azure Storage. This is a complete rewrite of > the previous WASB driver with a heavy focus on optimizing both performance > and cost. > {color:#212121} {color} > *{color:#212121}High level design{color}* > At a high level, the code here extends the FileSystem class to provide an > implementation for accessing blobs in Azure Storage. The scheme abfs is used > for accessing it over HTTP, and abfss for accessing over HTTPS. The following > URI scheme is used to address individual paths: > {color:#212121} {color} > > {color:#212121}abfs[s]://<filesystem>@<account>.dfs.core.windows.net/<path>{color} > {color:#212121} {color} > {color:#212121}ABFS is intended as a replacement to WASB. WASB is not > deprecated but is in pure maintenance mode and customers should upgrade to > ABFS once it hits General Availability later in CY18.{color} > {color:#212121}Benefits of ABFS include:{color} > {color:#212121}· Higher scale (capacity, throughput, and IOPS) Big > Data and Analytics workloads by allowing higher limits on storage > accounts{color} > {color:#212121}· Removing any ramp up time with Storage backend > partitioning; blocks are now automatically sharded across partitions in the > Storage backend{color} > {color:#212121} . This avoids the need for using > temporary/intermediate files, increasing the cost (and framework complexity > around committing jobs/tasks){color} > {color:#212121}· Enabling much higher read and write throughput on > single files (tens of Gbps by default){color} > {color:#212121}· Still retaining all of the Azure Blob features > customers are familiar with and expect, and gaining the benefits of future > Blob features as well{color} > {color:#212121}ABFS incorporates Hadoop Filesystem metrics to monitor the > file system throughput and operations. Ambari metrics are not currently > implemented for ABFS, but will be available soon.{color} > {color:#212121} {color} > *{color:#212121}Credits and history{color}* > Credit for this work goes to (hope I don't forget anyone): Shane Mainali, > {color:#212121}Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar > Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, > and James Baker. {color} > {color:#212121} {color} > *Test* > ABFS has gone through many test procedures including Hadoop file system > contract tests, unit testing, functional testing, and manual testing. All the > Junit tests provided with the driver are capable of running in both > sequential/parallel fashion in order to reduce the testing time. > {color:#212121}Besides unit tests, we have used ABFS as the default file > system in Azure HDInsight. Azure HDInsight will very soon offer ABFS as a > storage option. (HDFS is also used but not as default file system.) Various > different customer and test workloads have been run against clusters with > such configurations for quite some time. Benchmarks such as Tera*, TPC-DS, > Spark Streaming and Spark SQL, and others have been run to do scenario, > performance, and functional testing. Third parties and customers have also > done various testing of ABFS.{color} > {color:#212121}The current version reflects to the version of the code > tested and used in our production environment.{color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org