[
https://issues.apache.org/jira/browse/HADOOP-19604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18005100#comment-18005100
]
ASF GitHub Bot commented on HADOOP-19604:
-----------------------------------------
anujmodi2021 commented on code in PR #7777:
URL: https://github.com/apache/hadoop/pull/7777#discussion_r2204613657
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAppend.java:
##########
@@ -1041,16 +1041,14 @@ public void testWriteAsyncOpFailedAfterCloseCalled()
throws Exception {
/**
* Helper method that generates blockId.
- * @param position The offset needed to generate blockId.
* @return String representing the block ID generated.
*/
- private String generateBlockId(AbfsOutputStream os, long position) {
+ private String generateBlockId(AbfsOutputStream os) {
Review Comment:
Better to use production code in tests as well.
Any issue in code better to catch in production and fix there itself.
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChecksum.java:
##########
@@ -61,9 +64,36 @@ public class ITestAzureBlobFileSystemChecksum extends
AbstractAbfsIntegrationTes
private static final int MB_15 = 15 * ONE_MB;
private static final int MB_16 = 16 * ONE_MB;
private static final String INVALID_MD5_TEXT = "Text for Invalid MD5
Computation";
+ private MessageDigest md = null;
public ITestAzureBlobFileSystemChecksum() throws Exception {
super();
+ try {
+ md = MessageDigest.getInstance(MD5);
+ } catch (NoSuchAlgorithmException e) {
+ // MD5 algorithm not available; md will remain null
+ }
+ }
+
+ /**
+ * Computes the MD5 checksum of a specified portion of the input byte array.
+ *
+ * @param data The byte array containing the data to compute the MD5
checksum for.
+ * @param off The starting offset in the byte array.
+ * @param length The number of bytes to include in the checksum computation.
+ * @return The Base64-encoded MD5 checksum of the specified data, or null if
the digest is empty.
+ * @throws IllegalArgumentException If the offset or length is invalid for
the given byte array.
+ */
+ public String getMd5(byte[] data, int off, int length) {
Review Comment:
A similar method is present in production code.
Can we use that itself in tests so that production method can also be
covered in test flow?
If some issuei is found later then someone might just fix here and
production code will still remain buggy.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AzureIngressHandler.java:
##########
@@ -206,4 +208,32 @@ protected InvalidIngressServiceException
getIngressHandlerSwitchException(
* @return the block manager
*/
public abstract AbfsClient getClient();
+
+ /**
+ * Computes the Base64-encoded MD5 hash of the full blob content.
+ *
+ * <p>This method clones the current state of the {@link MessageDigest}
instance
+ * associated with the blob content to avoid resetting its original state.
It then
+ * calculates the MD5 digest and encodes it into a Base64 string.</p>
+ *
+ * @return A Base64-encoded string representing the MD5 hash of the full
blob content,
+ * or {@code null} if the digest could not be computed.
+ */
+ protected String computeFullBlobMd5() {
Review Comment:
Is there a unit test adde around this?
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestWasbAbfsCompatibility.java:
##########
@@ -201,4 +325,1650 @@ public void testSetWorkingDirectory() throws Exception {
assertEquals(path3, wasb.getWorkingDirectory());
assertEquals(path3, abfs.getWorkingDirectory());
}
+
+ // Scenario wise testing
+
+ //Scenario 1: - Create and write via WASB, read via ABFS
Review Comment:
Can we convert all these comments to javadocs for test methods also
inlcuding the expected outcome of eac scenario
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestWasbAbfsCompatibility.java:
##########
@@ -201,4 +325,1650 @@ public void testSetWorkingDirectory() throws Exception {
assertEquals(path3, wasb.getWorkingDirectory());
assertEquals(path3, abfs.getWorkingDirectory());
}
+
+ // Scenario wise testing
+
+ //Scenario 1: - Create and write via WASB, read via ABFS
+ @Test
+ public void testScenario1() throws Exception {
+ AzureBlobFileSystem abfs = getFileSystem();
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ // Write
+ try (FSDataOutputStream nativeFsStream = wasb.create(path, true)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+ // Check file status
+ ContractTestUtils.assertIsFile(wasb, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(abfs.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + abfs,
+ TEST_CONTEXT, line);
+ }
+
+ // Remove file
+ assertDeleted(abfs, path, true);
+ }
+
+ //Scenario 2: - Create and write via WASB, read via ABFS and then write the
same file via ABFS
+ @Test
+ public void testScenario2() throws Exception {
+ AzureBlobFileSystem abfs = getFileSystem();
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ assumeBlobServiceType();
+ Assume.assumeFalse("Not valid for APPEND BLOB", isAppendBlobEnabled());
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ // Write
+ try (FSDataOutputStream nativeFsStream = wasb.create(path, true)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+ // Check file status
+ ContractTestUtils.assertIsFile(wasb, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(abfs.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + abfs,
+ TEST_CONTEXT, line);
+ }
+
+ // Write
+ try (FSDataOutputStream abfsOutputStream = abfs.append(path)) {
+ abfsOutputStream.write(TEST_CONTEXT1.getBytes());
+ abfsOutputStream.flush();
+ abfsOutputStream.hsync();
+ }
+
+ // Remove file
+ assertDeleted(abfs, path, true);
+ }
+
+ //Scenario 3: - Create and write via ABFS and the read via WASB
+ @Test
+ public void testScenario3() throws Exception {
+ AzureBlobFileSystem abfs = getFileSystem();
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ Assume.assumeFalse("Not valid for APPEND BLOB", isAppendBlobEnabled());
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ // Write
+ try (FSDataOutputStream nativeFsStream = abfs.create(path, true)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+
+ // Check file status
+ ContractTestUtils.assertIsFile(abfs, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(wasb.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + wasb,
+ TEST_CONTEXT, line);
+ }
+ // Remove file
+ assertDeleted(abfs, path, true);
+ }
+
+ //Scenario 4:- Create via WASB, write via ABFS and then write via WASB
+ @Test
+ public void testScenario4() throws Exception {
+ AzureBlobFileSystem abfs = getFileSystem();
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ assumeBlobServiceType();
+ Assume.assumeFalse("Not valid for APPEND BLOB", isAppendBlobEnabled());
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ // Write
+ wasb.create(path, true);
+ try (FSDataOutputStream abfsOutputStream = abfs.append(path)) {
+ abfsOutputStream.write(TEST_CONTEXT.getBytes());
+ abfsOutputStream.flush();
+ abfsOutputStream.hsync();
+ }
+
+ try (FSDataOutputStream nativeFsStream = wasb.append(path)) {
+ nativeFsStream.write(TEST_CONTEXT1.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+
+ // Check file status
+ ContractTestUtils.assertIsFile(abfs, path);
+ // Remove file
+ assertDeleted(abfs, path, true);
+ }
+
+ //Scenario 5:- Create via ABFS, write via WASB, read via ABFS (Checksum
validation disabled)
+ @Test
+ public void testScenario5() throws Exception {
+ Configuration conf = getRawConfiguration();
+ conf.setBoolean(FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, false);
+ FileSystem fileSystem = FileSystem.newInstance(conf);
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fileSystem;
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ assumeBlobServiceType();
+ Assume.assumeFalse("Not valid for APPEND BLOB", isAppendBlobEnabled());
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ // Write
+ abfs.create(path, true);
+ try (FSDataOutputStream nativeFsStream = wasb.append(path)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+
+ // Check file status
+ ContractTestUtils.assertIsFile(abfs, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(abfs.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + abfs,
+ TEST_CONTEXT, line);
+ }
+
+ // Remove file
+ assertDeleted(abfs, path, true);
+ }
+
+ //Scenario 6: - Create via ABFS, write via WASB, read via ABFS (Checksum
validation enabled)
+ @Test
+ public void testScenario6() throws Exception {
+ Configuration conf = getRawConfiguration();
+ conf.setBoolean(FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, true);
+ FileSystem fileSystem = FileSystem.newInstance(conf);
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fileSystem;
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ Assume.assumeFalse("Not valid for APPEND BLOB", isAppendBlobEnabled());
+ assumeBlobServiceType();
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ // Write
+ abfs.create(path, true);
+ try (FSDataOutputStream nativeFsStream = wasb.append(path)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+
+ // Check file status
+ ContractTestUtils.assertIsFile(abfs, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(abfs.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + abfs,
+ TEST_CONTEXT, line);
+ }
+
+ // Remove file
+ assertDeleted(abfs, path, true);
+ }
+
+ // Scenario 7 :- Create via WASB and then create overwrite true using ABFS
+ @Test
+ public void testScenario7() throws Exception {
+ Configuration conf = getRawConfiguration();
+ conf.setBoolean(FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, true);
+ FileSystem fileSystem = FileSystem.newInstance(conf);
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fileSystem;
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ Assume.assumeFalse("Not valid for APPEND BLOB", isAppendBlobEnabled());
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ try (FSDataOutputStream nativeFsStream = wasb.create(path, true)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+
+ // Check file status
+ ContractTestUtils.assertIsFile(abfs, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(abfs.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + abfs,
+ TEST_CONTEXT, line);
+ }
+ abfs.create(path, true);
+
+ // Remove file
+ assertDeleted(abfs, path, true);
+ }
+
+ // Scenario 8 :- Create via WASB and then create overwrite false using ABFS
+ @Test
+ public void testScenario8() throws Exception {
+ Configuration conf = getRawConfiguration();
+ conf.setBoolean(FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, true);
+ FileSystem fileSystem = FileSystem.newInstance(conf);
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fileSystem;
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ Assume.assumeFalse("Not valid for APPEND BLOB", isAppendBlobEnabled());
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ try (FSDataOutputStream nativeFsStream = wasb.create(path, true)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+
+ // Check file status
+ ContractTestUtils.assertIsFile(abfs, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(abfs.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + abfs,
+ TEST_CONTEXT, line);
+ }
+ try {
+ abfs.create(path, false);
+ } catch (Exception e) {
+ assertTrue(e.getMessage().contains("AlreadyExists"));
+ }
+
+ // Remove file
+ assertDeleted(abfs, path, true);
+ }
+
+ // Scenario 9 :- Create via ABFS and then create overwrite true using WASB
+ @Test
+ public void testScenario9() throws Exception {
+ Configuration conf = getRawConfiguration();
+ conf.setBoolean(FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, true);
+ FileSystem fileSystem = FileSystem.newInstance(conf);
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fileSystem;
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ assumeBlobServiceType();
+ Assume.assumeFalse("Not valid for APPEND BLOB", isAppendBlobEnabled());
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ try (FSDataOutputStream nativeFsStream = abfs.create(path, true)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+
+ // Check file status
+ ContractTestUtils.assertIsFile(abfs, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(abfs.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + abfs,
+ TEST_CONTEXT, line);
+ }
+ wasb.create(path, true);
+
+ // Remove file
+ assertDeleted(abfs, path, true);
+ }
+
+ // Scenario 10 :- Create via ABFS and then create overwrite false using WASB
+ @Test
+ public void testScenario10() throws Exception {
+ Configuration conf = getRawConfiguration();
+ conf.setBoolean(FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, true);
+ FileSystem fileSystem = FileSystem.newInstance(conf);
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fileSystem;
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ try (FSDataOutputStream nativeFsStream = abfs.create(path, true)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+
+ // Check file status
+ ContractTestUtils.assertIsFile(abfs, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(abfs.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + abfs,
+ TEST_CONTEXT, line);
+ }
+ try {
+ wasb.create(path, false);
+ } catch (Exception e) {
+ assertTrue(e.getMessage().toLowerCase().contains("exists"));
+ }
+
+ // Remove file
+ assertDeleted(abfs, path, true);
+ }
+
+ // Scenario 11 :- Create via ABFS and then write via WASB and delete via ABFS
+ @Test
+ public void testScenario11() throws Exception {
+ Configuration conf = getRawConfiguration();
+ conf.setBoolean(FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, true);
+ FileSystem fileSystem = FileSystem.newInstance(conf);
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fileSystem;
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ assumeBlobServiceType();
+ Assume.assumeFalse("Not valid for APPEND BLOB", isAppendBlobEnabled());
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ // Write
+ abfs.create(path, true);
+ try (FSDataOutputStream nativeFsStream = wasb.append(path)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+
+ // Check file status
+ ContractTestUtils.assertIsFile(abfs, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(abfs.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + abfs,
+ TEST_CONTEXT, line);
+ }
+ abfs.delete(path, true);
+ }
+
+ // Scenario 12 :- Create and write via ABFS and delete via WASB
+ @Test
+ public void testScenario12() throws Exception {
+ Configuration conf = getRawConfiguration();
+ conf.setBoolean(FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, true);
+ FileSystem fileSystem = FileSystem.newInstance(conf);
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fileSystem;
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ // Write
+ try (FSDataOutputStream nativeFsStream = abfs.create(path, true)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+
+ // Check file status
+ ContractTestUtils.assertIsFile(abfs, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(abfs.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + abfs,
+ TEST_CONTEXT, line);
+ }
+ wasb.delete(path, true);
+ }
+
+ // Scenario 13:- Create via ABFS, write via WASB, and read via wasb
+ @Test
+ public void testScenario13() throws Exception {
+ Configuration conf = getRawConfiguration();
+ conf.setBoolean(FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, true);
+ FileSystem fileSystem = FileSystem.newInstance(conf);
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fileSystem;
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ assumeBlobServiceType();
+ Assume.assumeFalse("Not valid for APPEND BLOB", isAppendBlobEnabled());
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ // Write
+ abfs.create(path, true);
+ try (FSDataOutputStream nativeFsStream = wasb.append(path)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+
+ // Check file status
+ ContractTestUtils.assertIsFile(abfs, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(wasb.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + wasb,
+ TEST_CONTEXT, line);
+ }
+ abfs.delete(path, true);
+ }
+
+ // Scenario 14:- Create via ABFS, write via WASB, and delete via wasb
+ @Test
+ public void testScenario14() throws Exception {
+ Configuration conf = getRawConfiguration();
+ conf.setBoolean(FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, true);
+ FileSystem fileSystem = FileSystem.newInstance(conf);
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fileSystem;
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ assumeBlobServiceType();
+ Assume.assumeFalse("Not valid for APPEND BLOB", isAppendBlobEnabled());
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ // Write
+ abfs.create(path, true);
+ try (FSDataOutputStream nativeFsStream = wasb.append(path)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+
+ // Check file status
+ ContractTestUtils.assertIsFile(abfs, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(wasb.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + wasb,
+ TEST_CONTEXT, line);
+ }
+ wasb.delete(path, true);
+ }
+
+ // Scenario 15 :- Create and write via WASB and delete via ABFS
+ @Test
+ public void testScenario15() throws Exception {
+ Configuration conf = getRawConfiguration();
+ conf.setBoolean(FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, true);
+ FileSystem fileSystem = FileSystem.newInstance(conf);
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fileSystem;
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ // Write
+ try (FSDataOutputStream nativeFsStream = wasb.create(path, true)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+
+ // Check file status
+ ContractTestUtils.assertIsFile(abfs, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(wasb.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + wasb,
+ TEST_CONTEXT, line);
+ }
+ abfs.delete(path, true);
+ }
+
+ // Scenario 16: Create via WASB, write via ABFS, and delete via WASB
+ @Test
+ public void testScenario16() throws Exception {
+ Configuration conf = getRawConfiguration();
+ conf.setBoolean(FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, true);
+ FileSystem fileSystem = FileSystem.newInstance(conf);
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fileSystem;
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ assumeBlobServiceType();
+ Assume.assumeFalse("Not valid for APPEND BLOB", isAppendBlobEnabled());
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ // Write
+ wasb.create(path, true);
+ try (FSDataOutputStream nativeFsStream = abfs.append(path)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+
+ // Check file status
+ ContractTestUtils.assertIsFile(abfs, path);
+
+ try (BufferedReader br = new BufferedReader(
+ new InputStreamReader(abfs.open(path)))) {
+ String line = br.readLine();
+ assertEquals("Wrong text from " + abfs,
+ TEST_CONTEXT, line);
+ }
+ wasb.delete(path, true);
+ }
+
+ // Scenario 17: Create, setXAttr and getXAttr via ABFS
+ @Test
+ public void testScenario17() throws Exception {
+ Configuration conf = getRawConfiguration();
+ conf.setBoolean(FS_AZURE_ABFS_ENABLE_CHECKSUM_VALIDATION, true);
+ FileSystem fileSystem = FileSystem.newInstance(conf);
+ AzureBlobFileSystem abfs = (AzureBlobFileSystem) fileSystem;
+ Assume.assumeFalse("Namespace enabled account does not support this test",
+ getIsNamespaceEnabled(abfs));
+ NativeAzureFileSystem wasb = getWasbFileSystem();
+
+ Path testFile = path("/testReadFile");
+ Path path = new Path(testFile + "/~12/!008/testfile_" + UUID.randomUUID());
+
+ // Write
+ try (FSDataOutputStream nativeFsStream = abfs.create(path, true)) {
+ nativeFsStream.write(TEST_CONTEXT.getBytes());
+ nativeFsStream.flush();
+ nativeFsStream.hsync();
+ }
+ // -
> ABFS: Fix WASB ABFS compatibility issues
> ----------------------------------------
>
> Key: HADOOP-19604
> URL: https://issues.apache.org/jira/browse/HADOOP-19604
> Project: Hadoop Common
> Issue Type: Sub-task
> Affects Versions: 3.4.1
> Reporter: Anmol Asrani
> Assignee: Anmol Asrani
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.1
>
>
> Fix WASB ABFS compatibility issues. Fix issues such as:-
> # BlockId computation to be consistent across clients for PutBlock and
> PutBlockList
> # Restrict url encoding of certain json metadata during setXAttr calls.
> # Maintain the md5 hash of whole block to validate data integrity during
> flush.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
