Benoit Tellier created JAMES-4231:
-------------------------------------
Summary: S3 object compaction
Key: JAMES-4231
URL: https://issues.apache.org/jira/browse/JAMES-4231
Project: James Server
Issue Type: New Feature
Reporter: Benoit Tellier
h3. Why ?
Object store for small object is not optimal. For ~25KB it can represent up to
5% of the overall cost. On 600 TB of logical data this means operating on S3
side a 30 TB medatata DB, which can be huge.
This proposal is a current reflexion to try to mitigate this cost.
I think this can be proposed publically, but this *could* be implemented as a
Twake Mail feature first (dedicated objectstore).
h3. Key hypothesis and invariants
- blob are layed out in generation which is encoded in their blobId
- only live generation receives data.
- old generations are read + delete
- the likelyhood of a mail to be deleted is early in its lifecycle
- chuncks are immutable. We write new ones LSMT inspired
h3. desired properties
Deduplication is preserved
Objects are mutualized
Direct reads from the app remains possible without redirection
The process is fault tolerant. And transparent to the users.
The process makes progress (liveness property)
h3. What
Develop a notion of s3 compaction for old generation.
The idea is to group together in chunks of 100MB objects of less than 1MB. On
average a chunk is ~1000 objects.
We would then use a blobid wiich encode chunkid, limit, offset.
S3 metadata would need to be serialized in a header of the blob.
h3. the process
Initial compaction
{code:java}
1 -> Build a bloom filter over messagev3 to get an existence predicate
1.5 -> Build the temporary mapping table blobId -> messageid for this generation
2 -> Iterate over the non-compacted blobs and build the packs in RAM, computing
the offsets
3 -> Persist the pack to S3
3.5 -> Use the table from 1.5 to update the right tables
4 -> Delete the pre-existing objects
{code}
For gc-compact
{code:java}
1 -> Build a bloom filter over messagev3 to get an existence predicate
1.5 -> Build the temporary mapping table blobId -> messageid for this generation
2 -> Iterate over the non-compacted blobs, normal gc
2.5 -> Iterate over the packs, if needed purge and rewrite a pack (purge if 10%
dead), merge packs that are 50% dead
3 -> Persist the pack to S3
3.5 -> Use the table from 1.5 to update the right tables
4 -> Delete the old packs
{code}
In case of a crash, we trash 1.5 and the gc-compact will identify the
non-committed blocks and kill them. We get liveness by construction.
gc compact condition: at least 10% of size gain.
Note that we can attempt to merge chunck that are 50% below chunck size. Just
keep the chunk until we find another below 50% chunck size and merge. if above
50% commit or else keep going keeping it aside.
h3. complementary point of attention
Implemented in a side process not to impact live james
header blobs are present in messageidtable and imapuidtable in case of 404 we
can check the source of truth and fix in-place. ObjectNotFound exception can be
handled that way.
Chuncks start with a header that list all contained blobs (allowing GC to run)
h3. Chunck format
{code:java}
[format - byte]
[blob slot 1]
[blob slot 1]
[blob slot list footer - length list] -> allow knowing contained blobs
[footer position - long]
{code}
(negative Range: bytes=-65536 allow reading the footer)
blobslot list exemple: 2345~3456~4567 for chunckid1 means the following blobs
are contained:
- chunkid1~1~2345
- chunkid1~2346~3456
- etc...
Blob slot:
{code:java}
[contentstart] (long)
[CRC32C] -> body hash for integrity
[metadata]
[content]
{code}
with [metadata] :
{code:java}
content-encoding: zstd
content-original-size: 250000
{code}
To be directly addressable chuncks are not compressed only blob content of each
slot is. No Content-encoding either.
chunkId format [FAMILY int as string]_[GENERATION int as stream]_chunk_[random
128 bit value encoded as base64url no padding]
h3 source of inspiration
Dovecot pro object store bundling
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
