[ https://issues.apache.org/jira/browse/HDFS-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17231804#comment-17231804 ]
Ayush Saxena commented on HDFS-15680: ------------------------------------- Thanx [~ste...@apache.org] for volunteering. Always better to fix these things. I too had a very quick try. Something like this : {noformat} diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java index 1739cff76d4..e9429e7a848 100644 --- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java +++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java @@ -26,6 +26,7 @@ import java.net.HttpURLConnection; import java.net.URI; import java.net.URISyntaxException; +import java.net.URLEncoder; import java.util.ArrayList; import java.util.Calendar; import java.util.Date; @@ -37,6 +38,7 @@ import org.apache.commons.codec.DecoderException; import org.apache.commons.codec.net.URLCodec; import org.apache.commons.lang3.NotImplementedException; +import org.apache.hadoop.fs.Path; import org.apache.http.client.utils.URIBuilder; import com.microsoft.azure.storage.AccessCondition; @@ -137,7 +139,9 @@ private static String convertUriToDecodedString(URI uri) { private static URI convertKeyToEncodedUri(String key) { try { - return new URIBuilder().setPath(key).build(); + URI unEncodedURI = new URI(key); + return new URIBuilder().setPath(unEncodedURI.getRawPath()) + .setScheme(unEncodedURI.getScheme()).build(); } catch (URISyntaxException e) { throw new AssertionError("Failed to encode key: " + key); } @@ -148,7 +152,8 @@ public CloudBlobContainerWrapper getContainerReference(String name) throws URISyntaxException, StorageException { String fullUri; URIBuilder builder = new URIBuilder(baseUriString); - fullUri = builder.setPath(builder.getPath() + "/" + name).toString(); + String path = builder.getPath() == null ? "" : builder.getPath() + "/"; + fullUri = builder.setPath(path + name).toString(); MockCloudBlobContainerWrapper container = new MockCloudBlobContainerWrapper( fullUri, name); {noformat} I tried bunch of the tests, all apart from one {{testUriEncodingMoreComplexCharacters}} were passing. Guess need to have some other way to separate Scheme and Path. Give a check, if this can help you reach the right place. Well in case of any issues, though I am not very aware about this part of code, Still would be happy to help, in whichever way possible. :) > Disable Broken Azure Junits > --------------------------- > > Key: HDFS-15680 > URL: https://issues.apache.org/jira/browse/HDFS-15680 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs/azure > Reporter: Ahmed Hussein > Assignee: Ahmed Hussein > Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > There are 6 test classes have been failing on Yetus for several months. > They contributed to more than 41 failing tests which makes reviewing Yetus > reports every a pain in the neck. Another point is to save the resources and > avoiding utilization of ports, memory, and CPU. > Over the last month, there was some effort to bring the Yetus back to a > stable state. However, there is no progress in addressing Azure failures. > Generally, I do not like to disable failing tests, but for this specific > case, I do not assume that it makes any sense to have 41 failing tests from > one module for several months. Whenever someone finds that those tests are > useful, then they can re-enable the tests on Yetus *_After_* the test is > fixed. > Following a PR, I have to review that my patch does not cause any failures > (include changing error messages in existing tests). A thorough review takes > a considerable amount of time browsing the nightly builds and Github reports. > So, please consider how much time is being spent to review those stack trace > over the last months. > Finally, this is one of the reasons developers tend to ignore the reports, > because it would take too much time to review; and by default, the errors are > considered irrelevant. > CC: [~aajisaka], [~elgoiri], [~weichiu], [~ayushtkn] > {code:bash} > hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked > hadoop.fs.azure.TestNativeAzureFileSystemMocked > hadoop.fs.azure.TestBlobMetadata > hadoop.fs.azure.TestNativeAzureFileSystemConcurrency > hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck > hadoop.fs.azure.TestNativeAzureFileSystemContractMocked > hadoop.fs.azure.TestWasbFsck > hadoop.fs.azure.TestOutOfBandAzureBlobOperations > {code} > {code:bash} > org.apache.hadoop.fs.azure.TestBlobMetadata.testFolderMetadata > org.apache.hadoop.fs.azure.TestBlobMetadata.testFirstContainerVersionMetadata > org.apache.hadoop.fs.azure.TestBlobMetadata.testPermissionMetadata > org.apache.hadoop.fs.azure.TestBlobMetadata.testOldPermissionMetadata > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testNoTempBlobsVisible > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testLinkBlobs > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatusRootDir > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryMoveToExistingDirectory > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatus > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryAsExistingDirectory > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameToDirWithSamePrefixAllowed > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testLSRootDir > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testDeleteRecursively > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck.testWasbFsck > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testChineseCharactersFolderRename > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListing > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testUriEncoding > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testDeepFileCreation > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testListDirectory > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderRenameInProgress > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameFolder > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameImplicitFolder > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolder > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testStoreDeleteFolder > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRename > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testListStatus > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testRenameDirectoryAsEmptyDirectory > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testListStatusFilterWithSomeMatches > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testRenameDirectoryAsNonExistentDirectory > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testGlobStatusSomeMatchesInDirectories > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testGlobStatusWithMultipleWildCardMatches > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testDeleteRecursively > org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testImplicitFolderDeleted > org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testFileAndImplicitFolderSameName > org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testSetOwnerOnImplicitFolder > org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testFileInImplicitFolderDeleted > org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testImplicitFolderListed > org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testCreatingDeepFileCreatesExplicitFolder > org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations.testSetPermissionOnImplicitFolder > org.apache.hadoop.fs.azure.TestWasbFsck.testDelete > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org