[
https://issues.apache.org/jira/browse/HDFS-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18085178#comment-18085178
]
ASF GitHub Bot commented on HDFS-11161:
---------------------------------------
pan3793 commented on code in PR #8347:
URL: https://github.com/apache/hadoop/pull/8347#discussion_r3333161811
##########
hadoop-cloud-storage-project/hadoop-bos/src/test/resources/README-TEST.md:
##########
@@ -0,0 +1,326 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# BOS FileSystem Testing Guide
+
+This document provides guidance on running and understanding the BOS
FileSystem test suite.
+
+## Table of Contents
+
+1. [Prerequisites](#prerequisites)
+2. [Test Configuration](#test-configuration)
+3. [Running Tests](#running-tests)
+4. [Test Structure](#test-structure)
+5. [Troubleshooting](#troubleshooting)
+
+## Prerequisites
+
+### Required
+
+- Java 8 or higher
+- Maven 3.3 or higher
+- Access to a Baidu BOS bucket for testing
+- BOS credentials (Access Key ID and Secret Access Key)
+
+### BOS Bucket Setup
+
+1. Create a BOS bucket for testing (or use an existing one)
+2. Ensure the bucket is accessible with your credentials
+3. Note: Tests will create and delete files under `/test` directory in the
bucket
+
+## Test Configuration
+
+### Step 1: Create Test Configuration File
+
+Copy the template configuration file:
+
+```bash
+cd src/test/resources
+cp contract-test-options.xml.template contract-test-options.xml
+```
+
+### Step 2: Configure BOS Credentials
+
+Edit `contract-test-options.xml` and set your BOS configuration:
+
+```xml
+<property>
+ <name>fs.contract.test.fs.bos</name>
+ <value>bos://your-bucket-name/test</value>
+</property>
+
+<property>
+ <name>fs.bos.endpoint</name>
+ <value>http://bd.bcebos.com</value>
+</property>
+```
+
+### Step 3: Set Authentication
+
+**Recommended: Use Environment Variables**
+
+```bash
+export FS_BOS_ACCESS_KEY=your_access_key_id
+export FS_BOS_SECRET_ACCESS_KEY=your_secret_access_key
+# Optional: for STS/temporary credentials
+# export FS_BOS_SESSION_TOKEN_KEY=your_session_token
+```
+
+Ensure the credentials provider is set to
`EnvironmentVariableCredentialsProvider`:
+
+```xml
+<property>
+ <name>fs.bos.credentials.provider</name>
+
<value>org.apache.hadoop.fs.bos.credentials.EnvironmentVariableCredentialsProvider</value>
+</property>
+```
+
+**Alternative: Configuration-based (Not Recommended)**
+
+You can also configure credentials directly in `contract-test-options.xml`,
but this is **NOT recommended** for security reasons:
+
+```xml
+<property>
+ <name>fs.bos.access.key</name>
+ <value>YOUR_ACCESS_KEY</value>
+</property>
+
+<property>
+ <name>fs.bos.secret.key</name>
+ <value>YOUR_SECRET_KEY</value>
+</property>
+```
+
+**⚠️ SECURITY WARNING**: Never commit `contract-test-options.xml` with
credentials to version control!
+
+## Running Tests
+
+### Run All Tests
+
+```bash
+mvn test
+```
+
+### Run Contract Tests Only
+
+```bash
+mvn test -Dtest="**/contract/*Test.java"
+```
+
+### Run Specific Contract Test Suite
+
+```bash
+# Test file creation
+mvn test -Dtest=TestBosContractCreate
+
+# Test rename operations
+mvn test -Dtest=TestBosContractRename
+
+# Test delete operations
+mvn test -Dtest=TestBosContractDelete
+```
+
+### Run Integration Tests
+
+```bash
+mvn test -Dtest="**/integration/*Test.java"
+```
+
+### Run with Specific Bucket Type
+
+For namespace-enabled bucket:
+```bash
+mvn test -Dfs.contract.test.fs.bos=bos://namespace-bucket/test
+```
+
+For flat (non-namespace) bucket:
+```bash
+mvn test -Dfs.contract.test.fs.bos=bos://flat-bucket/test
+```
+
+### Skip Tests
+
+```bash
+mvn package -DskipTests
+```
+
+## Test Structure
+
+### Contract Tests (`org.apache.hadoop.fs.bos.contract`)
+
+These tests verify that BOS FileSystem conforms to the Hadoop FileSystem
contract:
+
+- **TestBosContractCreate**: File creation operations
+- **TestBosContractOpen**: File open and read operations
+- **TestBosContractDelete**: File and directory deletion
+- **TestBosContractMkdir**: Directory creation
+- **TestBosContractRename**: File and directory rename
+- **TestBosContractSeek**: Random access (seek) operations
+- **TestBosContractGetFileStatus**: Metadata operations
+- **TestBosContractRootDir**: Root directory operations (disabled by default)
+- **TestBosContractContentSummary**: Content summary operations
+
+### Integration Tests (`org.apache.hadoop.fs.bos.integration`)
+
+These tests verify BOS-specific functionality:
+
+- **TestBosIntegrationIO**: Input/Output stream operations
+- **TestBosIntegrationChecksum**: Checksum calculation and verification
+- **TestBosIntegrationMultipart**: Multi-part upload/download (if applicable)
+
+### Credential Tests (`org.apache.hadoop.fs.bos.credentials`)
+
+- **TestConfigurationCredentialsProvider**: Configuration-based authentication
+
+## Test Features and Limitations
+
+### Supported Features
+
+- ✅ File create, read, delete
+- ✅ Directory operations (mkdir, delete)
+- ✅ Rename (via copy + delete)
+- ✅ Random access (seek)
+- ✅ Content summary
+- ✅ Checksum verification
+
+### Unsupported Features (Tests Will Skip)
+
+- ❌ Append to files
+- ❌ File concatenation
+- ❌ File truncation
+- ❌ Symbolic links
+- ❌ Extended attributes (xattr)
+- ❌ Full Unix permissions (limited support in namespace mode)
+
Review Comment:
you seem to miss `hflush/hsync operations` item which you mentioned in
another place, and what's the behavior if the user calls `hflush/hsync`? noop
or throw an exception?
> Incorporate Baidu Yun BOS file system implementation
> ----------------------------------------------------
>
> Key: HDFS-11161
> URL: https://issues.apache.org/jira/browse/HDFS-11161
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: fs
> Reporter: Faen Zhang
> Priority: Major
> Labels: pull-request-available
> Original Estimate: 840h
> Remaining Estimate: 840h
>
> Baidu Yun ( https://cloud.baidu.com/ ) is one of top tier cloud computing
> provider. Baidu Yun BOS is widely used among China's cloud users, but
> currently it is not easy to access data laid on BOS storage from user's
> Hadoop/Spark application, because of no original support for BOS in Hadoop.
> This work aims to integrate Baidu Yun BOS with Hadoop. By simple
> configuration, Spark/Hadoop applications can read/write data from BOS without
> any code change. Narrowing the gap between user's APP and data storage, like
> what have been done for S3 and Aliyun OSS in Hadoop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
