Hi Andreas, AWS Lustre takes a snapshot of S3 file metadata on creation. However, I am trying to include files that are added to the S3 bucket after Lustre filesystem creation. Since I know which files were added, I wanted to simply inform the Lustre HSM layer of their existence. At this point I could unmount and re-create the filesystem, or I must copy the file from s3 to Lustre directly bypassing the HSM. Both of these options are pretty bad IMHO, since the HSM is more efficient and has better performance.
Kris On Wed, Jan 8, 2020 at 3:15 PM Andreas Dilger <adil...@whamcloud.com> wrote: > On Jan 8, 2020, at 06:13, Matt Rásó-Barnett <m...@rasobarnett.com> wrote: > > > Hi Kris, > I assume you are using Amazon's FSx for Lustre product ( > https://aws.amazon.com/fsx/lustre/) for this, rather than rolling Lustre > HSM to S3 on AWS yourself? > > I'm afraid I don't know any more than you do on this product but it's > something I've been keen to play with as it sounds really interesting. > > However the process or policy-engine by which Amazon are mapping S3 > objects to files in Lustre is not part of Lustre itself so you'd need input > from Amazon on this - maybe Aurelien (in BCC) can comment here? > > From their overview documenation my understanding is that the Lustre FS is > meant to be short lived in this model, so you create it populated with the > contents of the bucket *at that time*, run your workload, then archive > results back and delete the FS. > > So if your bucket has been updated post Lustre FS creation, you may have > to destroy the Lustre FS and recreate it to notice the changes. HSM isn't > designed to be a mechanism for synchronising two endpoints that change > independent of the other, so any change to the backend not via the > filesystem will not be tracked. > > > Wouldn't it be enough to "lfs hsm_release <file> ..." from Lustre, so that > Lustre drops the local copy of the file, and it is imported again from S3? > There would need to be some mechanism to run this command when the S3 > object is modified (preferably automatically), which is outside the scope > of Lustre. Are there S3 events that could be hooked in this case to > trigger an action, or a Changelog equivalent? > > Cheers, Andreas > > On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote: > > We have Lustre <- HSM -> S3 > > We have direct modifications to S3 that occur after the Lustre filesystem > is created > I was wondering if there is any way to register a new/deleted file at the > Lustre level using HSM or other commands > > Say a user uploads a file to S3, and I know the mapped path in Lustre, > I would like to do > lfs hsm_register /path/to/file/in/S3/ # Create a metadata entry in > Lustre > lfs hsm_restore /path/to/file/in/S3 # Fetch file from S3 into Lustre > > Thx > > > > > > > On Tue, Jan 7, 2020 at 8:04 AM Colin Faber <cfa...@gmail.com> wrote: > > Can you provide an example of what you're attempting to accomplish? Am I > understanding correctly, that you've got a lustre file system, you're then > writing data into this file system? > > On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval <k...@viqi.org> wrote: > > We are using Lustre on AWS backed by S3 buckets. > When creating a new Lustre filesystem, S3 metadata can be automatically > imported into Lustre. When changes occur to the underlying S3 store, > these changes are not automatically reflected. > > Is it possible to indicate the creation / deletion of the underlying S3 > files after filesystem creation using HSM? > Is it possible to reimport the underlying metadata after creation? > > Any pointers appreciated. > > Thanks, > Kris > > -- > Kris Kvilekval, Ph.D. > ViQi Inc > (805)-699-6081 > _______________________________________________ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > > > -- > Kris Kvilekval, Ph.D. > ViQi Inc > (805)-699-6081 > > > _______________________________________________ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > _______________________________________________ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > > Cheers, Andreas > -- > Andreas Dilger > Principal Lustre Architect > Whamcloud > > > > > > > _______________________________________________ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > -- Kris Kvilekval, Ph.D. ViQi Inc (805)-699-6081
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org