On Sep 06 2021, Moon silvery <silverymoon...@gmail.com> wrote:
> Hello,
>
> I love the design of S3QL so much that I want to use it in my homelab to
> replace my whole
> storage pool. But I have encountered a problem that when storing over 10
> million files in
> a mount point, S3QL creates so many objects in a single bucket, without any
> prefix to
> divide these files into sub-directories. This behavior is causing my object
> storage
> backend (self-hosted MINIO) creating over 10 million files under a single
> directory, which
> is very slow on any operation that need list folder (any filesystem could not
> handle so
> many files in a single directory).
> I have checked the document carefully, but I could not find any instruction
> on how could I
> change this behavior. What I want is let S3QL stores objects with prefix,
> like when it
> want to store s3ql_data_1197116, the directory structure would be
> s3ql_data_1/s3ql_data_1197/s3ql_data_1197116, so that any single directory
> would have no
> more than 1k files/sub-directories.
> Could you please help to take a look and share some idea? I know this could
> be a backend
> specific problem, that AWS or other cloud provider do not have this issue.
> But I indeed
> appreciate the design of S3QL and want to use it in every place I
> could. Thanks a lot!
Have you considered using the local:// S3QL backend instead of going
through Minio?
Best,
-Nikolaus
--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to s3ql+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/s3ql/87czpmt3l4.fsf%40vostro.rath.org.
