chibenwa opened a new pull request, #2710:
URL: https://github.com/apache/james-project/pull/2710
Avoids creating tons of intermediate folders thus reduces inode pressure
while still on disk splitting objects in 4.000 slots
So the MinIO retex gains complexity.
Actually we are encountering more complex than it seems issues with it.
After the OOM as everyting was stored in single GB big folder ( !!! ) that
is impossible to ls, thus breaking MinIO inner work (...) which needs spreading
accross several folders, we did run into a inode shortage issue (isize 512,
inpode space percentage 5%, XFS). Our study found that each object stored
resulted in ~5 inodes being used (including versionning, meta info part info,
object entry, but also `/`` into the part.
Email being small blob intensive, we are running out of into before running
out of disk space.
One generation with 4 `/` and storing ~2 million objects consummes 2,250 M
additional inodes with the folders prefix. Density of level 4 is terrible: 1
folder for one object. Density for level 3 is also not good: one level for ~8
underlying levels.
We thus believe that 2 levels (whish already splits objects in slots of 4096
items thus can scale to hundreds of millions items per generation) is thus a
much more default.
Please also find hereby recommended XFS settings / MinIO settings for a 4
node cluster:
- isize = 256 (which is XFS default and not 512)
- imaxpct = 10
- Errasure coding settings: 8 stripes with 2 or 3 parity block which allows
for a 75-63% density while tollerating one server failure (and potentially one
additional disk failure). Note choosing a strip of 16 is important as it
spreads workload better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
