prosgarz35 commented on PR #3197:
URL: https://github.com/apache/james-project/pull/3197#issuecomment-5792064937
# Detailed Test Suite & Verification Report: File BlobStore Folder Hierarchy
## Overview
This document provides a comprehensive breakdown of the test suite
implemented, verified, and used to validate the **File BlobStore Folder
Hierarchy** enhancement in Apache James.
Following review feedback from lead committer Benoit Tellier (`chibenwa`),
the solution leverages the existing folder hierarchy strategy
(`MinIOGenerationAwareBlobId`), which introduces `/` path separators into blob
identifiers:
```
<family>/<generation>/<c1>/<c2>/<remaining-entropy>
e.g.: 1/628/M/f/emXjFVhqwZi9eYtmKc5A
```
This architecture naturally creates nested directory structures directly
through `new File(bucketRoot, blobId.asString())`, avoiding directory bloat
(100+ million objects safely partitioned across subdirectories).
---
## 1. Test Suite Modules & Executed Classes
### Module: `server/blob/blob-file`
This module is the core implementation module for filesystem-based blob
storage. The full test suite was executed using Apache Maven and JDK 25:
```bash
mvn test -pl server/blob/blob-file
```
#### Results Summary:
* **Total Tests Run:** 339
* **Failures:** 0
* **Errors:** 0
* **Skipped:** 0
* **Build Status:** SUCCESS
---
## 2. Detailed Breakdown of Test Classes
### 2.1. `FileWithMinIOGenerationAwareBlobIdTest` (New Test Suite)
* **File:**
[`server/blob/blob-file/src/test/java/org/apache/james/blob/file/FileWithMinIOGenerationAwareBlobIdTest.java`](file:///C:/soft/git_blob/server/blob/blob-file/src/test/java/org/apache/james/blob/file/FileWithMinIOGenerationAwareBlobIdTest.java)
* **Description:** Dedicated contract and integration test suite asserting
`FileBlobStore` behavior when configured with
`MinIOGenerationAwareBlobId.Factory`. Mirrors
`S3WithMinIOGenerationAwareBlobIdTest` from `blob-s3`.
* **Total Tests:** 77 tests
#### Key Test Scenarios:
1. **Blob Storage Contract Compliance (`BlobStoreContract` /
`BucketBlobStoreContract`):**
* Verifies standard blob store operations: `save`, `read`, `readBytes`,
`readReactive`, `listBlobs`, `listBuckets`, `deleteBucket`.
* Covers all storage policies: `LOW_COST`, `SIZE_BASED`, and
`HIGH_PERFORMANCE`.
* Tests payloads of various sizes: empty byte array, short strings, 11 KB
streams, and large multi-megabyte streams (12 MB).
2. **Folder Hierarchy Formatting & Entropy Generation:**
* **Test:** `saveShouldReturnBlobIdOfString(BlobStore.StoragePolicy
storagePolicy)`
* **Assertion:**
```java
assertThat(blobIdString).isEqualTo("1/628/M/f/emXjFVhqwZi9eYtmKc5A");
assertThat(blobId).isEqualTo(blobIdFactory().parse(blobIdString));
```
* **Validation:** Validates that saving a blob creates the expected
generation prefix (`1/628/`) and the 2-level subfolder path (`M/f/`) with the
truncated base64url content-addressed hash.
3. **Pruning of Empty Parent Directories on Deletion:**
* **Test:** `deleteShouldPruneEmptyParentDirectories()`
* **Scenario:**
* Saves a blob with `MinIOGenerationAwareBlobId`.
* Verifies that the nested file `default/1/628/M/f/<hash>` and its
intermediate directories exist on disk.
* Invokes `fileBlobStoreDAO.delete(defaultBucketName, blobId)`.
* Asserts that `blobFile.doesNotExist()` and `parentDir.doesNotExist()`.
* **Importance:** Prevents intermediate directory accumulation when blobs
are purged or collected by GC.
4. **Backward & Forward Compatibility (`Nested class Compatible`):**
*
**`readWithMinIOGenerationAwareShouldSuccessWhenBlobWasStoredByGenerationAware`**:
Verifies that a blob written using legacy flat `GenerationAwareBlobId`
(`1_628_<hash>`) can be read seamlessly by a store running
`MinIOGenerationAwareBlobId`.
*
**`listBlobsShouldReturnCorrectBlobIdWhenBlobWasStoredByGenerationAware`**:
Verifies that `listBlobs()` returns the correct `BlobId` instance for
legacy blobs, allowing data to be read without loss.
*
**`readWithGenerationAwareShouldSuccessWhenBlobWasStoredByMinIOGenerationAware`**:
Verifies that a store configured with `GenerationAwareBlobId` can parse
and read hierarchical blobs (`1/628/M/f/...`).
*
**`listBlobsShouldReturnCorrectBlobIdWhenBlobWasStoredByMinIOGenerationAware`**:
Verifies cross-store listing compatibility between hierarchical and
flat representations.
---
### 2.2. `FileBlobStoreTest` (Existing Regression Suite)
* **File:**
[`server/blob/blob-file/src/test/java/org/apache/james/blob/file/FileBlobStoreTest.java`](file:///C:/soft/git_blob/server/blob/blob-file/src/test/java/org/apache/james/blob/file/FileBlobStoreTest.java)
* **Description:** Standard deduplication file blob store suite implementing
`MetricableBlobStoreContract` and `DeduplicationBlobStoreContract`.
* **Total Tests:** 85 tests (All passed)
* **Coverage:** Validates deduplication, metrics collection, stream
read/write pipelines, and basic blob store behaviors with `PlainBlobId`.
---
### 2.3. `FileBlobStorePassThroughTest` (Existing Regression Suite)
* **File:**
`server/blob/blob-file/src/test/java/org/apache/james/blob/file/FileBlobStorePassThroughTest.java`
* **Description:** Tests the `PassThroughBlobStore` strategy on top of
`FileBlobStoreDAO`.
* **Total Tests:** 86 tests (All passed)
* **Coverage:** Validates that passthrough mode without deduplication
functions reliably without collisions.
---
### 2.4. `FileBlobStoreGCAlgorithmTest` (Garbage Collection Suite)
* **File:**
[`server/blob/blob-file/src/test/java/org/apache/james/blob/file/FileBlobStoreGCAlgorithmTest.java`](file:///C:/soft/git_blob/server/blob/blob-file/src/test/java/org/apache/james/blob/file/FileBlobStoreGCAlgorithmTest.java)
* **Description:** Verifies Bloom filter GC algorithm contracts
(`BloomFilterGCAlgorithmContract`) using `FileBlobStoreDAO`.
* **Total Tests:** 23 tests (All passed)
* **Coverage:** Generation awareness, orphan detection, reference tracking,
and multiple GC cycles.
---
### 2.5. `FileBlobStoreDAOTest` (DAO Unit Suite)
* **File:**
[`server/blob/blob-file/src/test/java/org/apache/james/blob/file/FileBlobStoreDAOTest.java`](file:///C:/soft/git_blob/server/blob/blob-file/src/test/java/org/apache/james/blob/file/FileBlobStoreDAOTest.java)
* **Description:** Unit and integration contract tests for `BlobStoreDAO`
and extended file metadata attribute operations
(`MetadataAwareBlobStoreDAOContract`).
* **Total Tests:** 68 tests (All passed)
* **Coverage:** User-defined file attribute views (XATTRs), reactive and
blocking streams, atomic file movement via temporary files, and bucket deletion
idempotency.
---
## 3. Module: `server/container/guice/blob/deduplication-gc`
* **File Modified:**
[`server/container/guice/blob/deduplication-gc/src/main/java/org/apache/james/modules/blobstore/BlobDeduplicationGCModule.java`](file:///C:/soft/git_blob/server/container/guice/blob/deduplication-gc/src/main/java/org/apache/james/modules/blobstore/BlobDeduplicationGCModule.java)
* **Compilation & Checkstyle Status:**
```bash
mvn test-compile -pl server/container/guice/blob/deduplication-gc
```
* `BUILD SUCCESS` (10 goals executed, 0 checkstyle violations).
* **Logic Verified:**
Guice provider `generationAwareBlobIdFactory` resolves the active mode
using the prioritized fallback:
```java
boolean compatibilityModeActivated =
Optional.ofNullable(System.getProperty("james.blobstore.folder.hierarchy"))
.or(() ->
Optional.ofNullable(System.getProperty("james.s3.minio.compatibility.mode")))
.map(Boolean::parseBoolean)
.orElse(false);
```
Returns `MinIOGenerationAwareBlobId.Factory` when enabled, or
`GenerationAwareBlobId.Factory` when disabled.
---
## 4. Configuration & Documentation Verification
### 4.1. Sample Configuration (`server/apps/postgres-app`)
* **File:**
[`server/apps/postgres-app/sample-configuration/jvm.properties`](file:///C:/soft/git_blob/server/apps/postgres-app/sample-configuration/jvm.properties)
* **Added Property:**
```properties
# Enable folder hierarchy across subdirectories for blob storage (S3/MinIO
and FileBlobStore) to avoid directory bloat
james.blobstore.folder.hierarchy=true
```
### 4.2. Documentation (`docs/modules/servers`)
* **File:**
[`docs/modules/servers/partials/configure/blobstore.adoc`](file:///C:/soft/git_blob/docs/modules/servers/partials/configure/blobstore.adoc)
* **Anchor Preserved:** `[[_improve_listing_support_for_minio]]` to
guarantee that existing URLs and permalinks (such as
`blobstore.html#_improve_listing_support_for_minio`) remain valid.
* **Content:** Accurately documents folder hierarchy benefits for both MinIO
and `FileBlobStore`, the new property alias, default enablement in the Postgres
distribution, and backward-compatible legacy property support.
---
## 5. Summary Matrix
| Test Suite / Area | Tests Run | Result | Key Capabilities Verified |
| :--- | :---: | :---: | :--- |
| `FileWithMinIOGenerationAwareBlobIdTest` | 77 | **PASS** |
Sharding/hierarchy formatting, parent pruning, cross-compatibility |
| `FileBlobStoreTest` | 85 | **PASS** | Deduplication, content hashing,
metric wrappers |
| `FileBlobStorePassThroughTest` | 86 | **PASS** | Passthrough streaming
without deduplication |
| `FileBlobStoreDAOTest` | 68 | **PASS** | Atomic replace, XATTR metadata,
bucket lifecycle |
| `FileBlobStoreGCAlgorithmTest` | 23 | **PASS** | Bloom filter garbage
collection cycles |
| **Total `blob-file` Suite** | **339** | **PASS** | **100% test pass rate**
|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
