[
https://issues.apache.org/jira/browse/JAMES-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116947#comment-18116947
]
Hesanda Liyanage commented on JAMES-4231:
-----------------------------------------
Thank you!!! I'll start working on it 🤝🏻
> S3 object compaction
> --------------------
>
> Key: JAMES-4231
> URL: https://issues.apache.org/jira/browse/JAMES-4231
> Project: James Server
> Issue Type: New Feature
> Reporter: Benoit Tellier
> Priority: Major
>
> h3. Why ?
> Object store for small object is not optimal. For ~25KB it can represent up
> to 5% of the overall cost. On 600 TB of logical data this means operating on
> S3 side a 30 TB medatata DB, which can be huge.
> This proposal is a current reflexion to try to mitigate this cost.
> I think this can be proposed publically, but this *could* be implemented as a
> Twake Mail feature first (dedicated objectstore).
> h3. Key hypothesis and invariants
> - blob are layed out in generation which is encoded in their blobId
> - only live generation receives data.
> - old generations are read + delete
> - the likelyhood of a mail to be deleted is early in its lifecycle
> - chuncks are immutable. We write new ones LSMT inspired
> h3. desired properties
> Deduplication is preserved
> Objects are mutualized
> Direct reads from the app remains possible without redirection
> The process is fault tolerant. And transparent to the users.
> The process makes progress (liveness property)
> h3. What
> Develop a notion of s3 compaction for old generation.
> The idea is to group together in chunks of 100MB objects of less than 1MB. On
> average a chunk is ~1000 objects.
> We would then use a blobid wiich encode chunkid, limit, offset.
> S3 metadata would need to be serialized in a header of the blob.
> h3. the process
> Initial compaction
> {code:java}
> 1 -> Build a bloom filter over messagev3 to get an existence predicate
> 1.5 -> Build the temporary mapping table blobId -> messageid for this
> generation
> 2 -> Iterate over the non-compacted blobs and build the packs in RAM,
> computing the offsets
> 3 -> Persist the pack to S3
> 3.5 -> Use the table from 1.5 to update the right tables
> 4 -> Delete the pre-existing objects
> {code}
> For gc-compact
> {code:java}
> 1 -> Build a bloom filter over messagev3 to get an existence predicate
> 1.5 -> Build the temporary mapping table blobId -> messageid for this
> generation
> 2 -> Iterate over the non-compacted blobs, normal gc
> 2.5 -> Iterate over the packs, if needed purge and rewrite a pack (purge if
> 10% dead), merge packs that are 50% dead
> 3 -> Persist the pack to S3
> 3.5 -> Use the table from 1.5 to update the right tables
> 4 -> Delete the old packs
> {code}
> In case of a crash, we trash 1.5 and the gc-compact will identify the
> non-committed blocks and kill them. We get liveness by construction.
> gc compact condition: at least 10% of size gain.
> Note that we can attempt to merge chunck that are 50% below chunck size. Just
> keep the chunk until we find another below 50% chunck size and merge. if
> above 50% commit or else keep going keeping it aside.
> h3. complementary point of attention
> Implemented in a side process not to impact live james
> header blobs are present in messageidtable and imapuidtable in case of 404 we
> can check the source of truth and fix in-place. ObjectNotFound exception can
> be handled that way.
> Chuncks start with a header that list all contained blobs (allowing GC to run)
> h3. Chunck format
> {code:java}
> [format - byte]
> [blob slot 1]
> [blob slot 1]
> [blob slot list footer - length list] -> allow knowing contained blobs
> [footer position - long]
> {code}
> (negative Range: bytes=-65536 allow reading the footer)
> blobslot list exemple: 2345~3456~4567 for chunckid1 means the following blobs
> are contained:
> - chunkid1~1~2345
> - chunkid1~2346~3456
> - etc...
> Blob slot:
> {code:java}
> [contentstart] (long)
> [CRC32C] -> body hash for integrity
> [metadata]
> [content]
> {code}
> with [metadata] :
> {code:java}
> content-encoding: zstd
> content-original-size: 250000
> {code}
> To be directly addressable chuncks are not compressed only blob content of
> each slot is. No Content-encoding either.
> chunkId format [FAMILY int as string]_[GENERATION int as
> stream]_chunk_[random 128 bit value encoded as base64url no padding]
> h3 source of inspiration
> Dovecot pro object store bundling
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
