** Changed in: mahara Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Mahara Contributors, which is subscribed to Mahara. Matching subscriptions: Subscription for all Mahara Contributors -- please ask on #mahara-dev or mahara.org forum before editing or unsubscribing it! https://bugs.launchpad.net/bugs/1685049
Title: Modifications to filesystem to allow object storage Status in Mahara: Fix Released Bug description: To use object storage (like S3) several modifications are needed (like get_path should return either local path to file or remote if it is stored remotely). Use cases: Offloading large and old files to save money Disk can be expensive, so a simple use case is we simply want to move some of the largest and oldest files off local disk to somewhere cheaper. But we still want the convenience and performance of having the majority of files local, especially if you are hosting on-prem where the latency or bandwidth to the remote filesystem may not be great. Sharing files across maharas to save disk Clients can have multiple mahara instances, and there is much duplicated content across instances. By pointing multiple maharas at the same remote filesystem, and not allowing deletes, then large amounts of content can be de-duplicated. Sharing files across environments to save time We can have multiple environments for various types of testing, and often have ad hoc environments created on demand. Not only do we not want to have to store duplicated files, but we also want refreshing data to new environments to be as fast as possible. Using this plugin we can configure production to have full read write to the remote filesystem and store the vast bulk of content remotely. In this setup the latency and bandwidth isn't an issue as they are colocated. The local filedir on disk would only consist of small or fast churning files. A refresh of the production data back to a staging environment can be much quicker now as we skip the sitedir clone completely and stage is simple configured with readonly access to the production filesystem. Any files it creates would only be writen to it's local filesystem which can then be discarded when next refreshed. Currently it only works with AWS S3 storage. There is support for more object stores planed, in particular enabling Openstack deployments. To use a plugin you will need to create an Amazon S3 bucket for your mahara instance. You will also need Amazon SDK to make this plugin work. So currently following directory is being saved to S3 - /path_to_sitedata/artefact/file/originals There is a cron task that checks if there are new files that are not yet in S3 (depending on setting it only pushes files of certain age and size). Once files are duplicated, there is an option to delete local copy if we want to save space (cron task checks duplicated files, again only deletes those depending on setting, like large files etc). Certain operations only can deal only with local files - for example on content page - download content as zip archive. So when hitting download button, first all files will be pulled to local from S3. To manage notifications about this bug go to: https://bugs.launchpad.net/mahara/+bug/1685049/+subscriptions _______________________________________________ Mailing list: https://launchpad.net/~mahara-contributors Post to : mahara-contributors@lists.launchpad.net Unsubscribe : https://launchpad.net/~mahara-contributors More help : https://help.launchpad.net/ListHelp