Hi Baptiste, On 18/09/22 at 11:29 +0200, Baptiste Beauplat wrote: > Having files stored into an object-like storage could improve cost and > performence over the conventional storage that snapshot is currently > using. (Currently using over 130 TB!) > > Multiple componants needs to access snapshot farm. The importer, the web > app (if not redirected) and multiple other scripts. > > We should write a generic file driver to allow all those component to > access/update/delete file from a config-defined backend. > > This driver would be usable in at least two langauges: ruby and python. > I'm not sure what is the best course of action here. Some kind of > bindings or maintaining two separate drivers. > > Note that there is also some C program as part for snapshot (the fsck > program). > > I was thinking for writing at least two backend for the driver: > > - a standard flat filesystem storage (what we have currently) > - an object-like storage. S3 would be a good candidate since a couple of > opensource storage solution provide S3 compatible API. > > That would allow a two step transision: start using the driver, then > switch the backend.
I was wondering if you made some progress on this? Your plan looks very good. I agree that a S3 backend would make a lot of sense (usable both with self-hosted solutions like MinIO, or with managed services). Let me know if I can help somehow. Lucas
