Re: [lustre-discuss] reflecting state of underlying store in Lustre with HSM

[Matt Rásó-Barnett]

Hi Kris,
I assume you are using Amazon's FSx for Lustre product 
(https://aws.amazon.com/fsx/lustre/) for this, rather than rolling 
Lustre HSM to S3 on AWS yourself?


I'm afraid I don't know any more than you do on this product but it's 
something I've been keen to play with as it sounds really interesting.


However the process or policy-engine by which Amazon are mapping S3 
objects to files in Lustre is not part of Lustre itself so you'd need 
input from Amazon on this - maybe Aurelien (in BCC) can comment here?


From their overview documenation my understanding is that the Lustre FS 
is meant to be short lived in this model, so you create it populated 
with the contents of the bucket *at that time*, run your workload, then 
archive results back and delete the FS.


So if your bucket has been updated post Lustre FS creation, you may have 
to destroy the Lustre FS and recreate it to notice the changes. HSM 
isn't designed to be a mechanism for synchronising two endpoints that 
change independent of the other, so any change to the backend not via 
the filesystem will not be tracked.


This is just me guessing without having used it however, I'll be 
interested to hear if you learn more about this from Amazon.


Kind regards,

Matt

On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote:

We have Lustre <- HSM -> S3

We have direct modifications to S3 that occur after the Lustre filesystem
is created
I was  wondering if there is any way to register a new/deleted file at the
Lustre level using HSM or other commands

Say a user uploads a file to S3, and I know  the mapped path in Lustre,
I would like to do
lfs hsm_register /path/to/file/in/S3/ # Create a metadata entry in
Lustre
lfs hsm_restore /path/to/file/in/S3   # Fetch file from S3 into Lustre

Thx






On Tue, Jan 7, 2020 at 8:04 AM Colin Faber  wrote:


Can you provide an example of what you're attempting to accomplish?  Am I
understanding correctly, that you've got a lustre file system, you're then
writing data into this file system?

On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval  wrote:


We are using Lustre on AWS backed by S3 buckets.
When creating a new Lustre filesystem, S3 metadata can be automatically
imported  into Lustre.  When changes occur to the underlying S3 store,
these changes are not automatically reflected.

Is it possible to indicate the creation / deletion of the underlying S3
files after filesystem creation using HSM?
Is it possible to reimport the underlying metadata after creation?

Any pointers appreciated.

Thanks,
Kris

--
Kris Kvilekval, Ph.D.
ViQi Inc
(805)-699-6081
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org





--
Kris Kvilekval, Ph.D.
ViQi Inc
(805)-699-6081



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] reflecting state of underlying store in Lustre with HSM

[Thomas Stibor]

Hi there,

we use a similar approach with the TSM Lustre copytool. 
First the data is archived to a TSM Server, then one can do the
following:

$ lfs hsm_set --exists --archived --archive-id  
$ lfs hsm_release 

The file  now exists as a released file in Lustre and is
seamlessly retrieved when the bulk data is accessed on Lustre. Note, the 
the copytool "knows" the mapped path, in our TSM case (TSM object name
-> Lustre filename and vice versa)

e.g.: /fs/hl/ll -> /lustre/hl/ll

Here is e.g. a script to sync TSM object data to a Lustre filesystem via
the copytool:

https://raw.githubusercontent.com/tstibor/ltsm/master/script/ltsmsync.sh

Cheers
 Thomas

On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote:
> We have Lustre <- HSM -> S3
> 
> We have direct modifications to S3 that occur after the Lustre filesystem
> is created
> I was  wondering if there is any way to register a new/deleted file at the
> Lustre level using HSM or other commands
> 
> Say a user uploads a file to S3, and I know  the mapped path in Lustre,
> I would like to do
> lfs hsm_register /path/to/file/in/S3/ # Create a metadata entry in
> Lustre
> lfs hsm_restore /path/to/file/in/S3   # Fetch file from S3 into Lustre
> 
> Thx
> 
> 
> 
> 
> 
> 
> On Tue, Jan 7, 2020 at 8:04 AM Colin Faber  wrote:
> 
> > Can you provide an example of what you're attempting to accomplish?  Am I
> > understanding correctly, that you've got a lustre file system, you're then
> > writing data into this file system?
> >
> > On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval  wrote:
> >
> >> We are using Lustre on AWS backed by S3 buckets.
> >> When creating a new Lustre filesystem, S3 metadata can be automatically
> >> imported  into Lustre.  When changes occur to the underlying S3 store,
> >> these changes are not automatically reflected.
> >>
> >> Is it possible to indicate the creation / deletion of the underlying S3
> >> files after filesystem creation using HSM?
> >> Is it possible to reimport the underlying metadata after creation?
> >>
> >> Any pointers appreciated.
> >>
> >> Thanks,
> >> Kris
> >>
> >> --
> >> Kris Kvilekval, Ph.D.
> >> ViQi Inc
> >> (805)-699-6081
> >> ___
> >> lustre-discuss mailing list
> >> lustre-discuss@lists.lustre.org
> >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> >>
> >
> 
> -- 
> Kris Kvilekval, Ph.D.
> ViQi Inc
> (805)-699-6081

> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] reflecting state of underlying store in Lustre with HSM

[quentin.bouget]

Hi everyone,

I don't know much about Amazon's FSx for Lustre, nor about any S3 
copytool. But I do know that lhsmtool_posix (the POSIX copytool shipped 
with the Lustre sources), supports an "import" operation comparable to 
the "lfs hsm_register" Kris describes.


Actually, Amazon's FSx for Lustre probably sets up Lustre filesystems 
with something like:


for each file in S3 bucket:
    S3-copytool import "$file" into /mnt/lustre

@Kris, I think you should ask Amazon's support if they provide a 
dedicated tool that does what you want.


Otherwise, **if you are not afraid of shooting yourself in the foot**, 
you can have a look at "llapi_hsm_import()" in 
lustre/utils/liblustreapi_hsm.c 
. 
It might be just what you are looking for.


Quentin

On 08/01/2020 14:13, Matt Rásó-Barnett wrote:


Hi Kris,
I assume you are using Amazon's FSx for Lustre product 
(https://aws.amazon.com/fsx/lustre/) for this, rather than rolling 
Lustre HSM to S3 on AWS yourself?


I'm afraid I don't know any more than you do on this product but it's 
something I've been keen to play with as it sounds really interesting.


However the process or policy-engine by which Amazon are mapping S3 
objects to files in Lustre is not part of Lustre itself so you'd need 
input from Amazon on this - maybe Aurelien (in BCC) can comment here?


From their overview documenation my understanding is that the Lustre 
FS is meant to be short lived in this model, so you create it 
populated with the contents of the bucket *at that time*, run your 
workload, then archive results back and delete the FS.


So if your bucket has been updated post Lustre FS creation, you may 
have to destroy the Lustre FS and recreate it to notice the changes. 
HSM isn't designed to be a mechanism for synchronising two endpoints 
that change independent of the other, so any change to the backend not 
via the filesystem will not be tracked.


This is just me guessing without having used it however, I'll be 
interested to hear if you learn more about this from Amazon.


Kind regards,

Matt

On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote:

We have Lustre <- HSM -> S3

We have direct modifications to S3 that occur after the Lustre 
filesystem

is created
I was  wondering if there is any way to register a new/deleted file 
at the

Lustre level using HSM or other commands

Say a user uploads a file to S3, and I know  the mapped path in Lustre,
I would like to do
lfs hsm_register /path/to/file/in/S3/ # Create a metadata entry in
Lustre
lfs hsm_restore /path/to/file/in/S3   # Fetch file from S3 into 
Lustre


Thx






On Tue, Jan 7, 2020 at 8:04 AM Colin Faber  wrote:

Can you provide an example of what you're attempting to accomplish?  
Am I
understanding correctly, that you've got a lustre file system, 
you're then

writing data into this file system?

On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval  
wrote:



We are using Lustre on AWS backed by S3 buckets.
When creating a new Lustre filesystem, S3 metadata can be 
automatically

imported  into Lustre.  When changes occur to the underlying S3 store,
these changes are not automatically reflected.

Is it possible to indicate the creation / deletion of the 
underlying S3

files after filesystem creation using HSM?
Is it possible to reimport the underlying metadata after creation?

Any pointers appreciated.

Thanks,
Kris

--
Kris Kvilekval, Ph.D.
ViQi Inc
(805)-699-6081
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org





--
Kris Kvilekval, Ph.D.
ViQi Inc
(805)-699-6081



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] reflecting state of underlying store in Lustre with HSM

[Degremont, Aurelien]

Hi Kris,

As people said each HSM has its own copytool with its own constraints. To work 
properly with S3, additional metadata is stored in Lustre when files are 
imported from a S3 bucket.
Questions specific to Amazon FSx For Lustre would be better asked to AWS 
support. 

Aurélien

Le 08/01/2020 14:14, « Matt Rásó-Barnett »  a écrit :

Hi Kris,
I assume you are using Amazon's FSx for Lustre product 
(https://aws.amazon.com/fsx/lustre/) for this, rather than rolling 
Lustre HSM to S3 on AWS yourself?

I'm afraid I don't know any more than you do on this product but it's 
something I've been keen to play with as it sounds really interesting.

However the process or policy-engine by which Amazon are mapping S3 
objects to files in Lustre is not part of Lustre itself so you'd need 
input from Amazon on this - maybe Aurelien (in BCC) can comment here?

 From their overview documenation my understanding is that the Lustre FS 
is meant to be short lived in this model, so you create it populated 
with the contents of the bucket *at that time*, run your workload, then 
archive results back and delete the FS.

So if your bucket has been updated post Lustre FS creation, you may have 
to destroy the Lustre FS and recreate it to notice the changes. HSM 
isn't designed to be a mechanism for synchronising two endpoints that 
change independent of the other, so any change to the backend not via 
the filesystem will not be tracked.

This is just me guessing without having used it however, I'll be 
interested to hear if you learn more about this from Amazon.

Kind regards,

Matt

On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote:
>We have Lustre <- HSM -> S3
>
>We have direct modifications to S3 that occur after the Lustre filesystem
>is created
>I was  wondering if there is any way to register a new/deleted file at the
>Lustre level using HSM or other commands
>
>Say a user uploads a file to S3, and I know  the mapped path in Lustre,
>I would like to do
>lfs hsm_register /path/to/file/in/S3/ # Create a metadata entry in
>Lustre
>lfs hsm_restore /path/to/file/in/S3   # Fetch file from S3 into Lustre
>
>Thx
>
>
>
>
>
>
>On Tue, Jan 7, 2020 at 8:04 AM Colin Faber  wrote:
>
>> Can you provide an example of what you're attempting to accomplish?  Am I
>> understanding correctly, that you've got a lustre file system, you're 
then
>> writing data into this file system?
>>
>> On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval  wrote:
>>
>>> We are using Lustre on AWS backed by S3 buckets.
>>> When creating a new Lustre filesystem, S3 metadata can be automatically
>>> imported  into Lustre.  When changes occur to the underlying S3 store,
>>> these changes are not automatically reflected.
>>>
>>> Is it possible to indicate the creation / deletion of the underlying S3
>>> files after filesystem creation using HSM?
>>> Is it possible to reimport the underlying metadata after creation?
>>>
>>> Any pointers appreciated.
>>>
>>> Thanks,
>>> Kris
>>>
>>> --
>>> Kris Kvilekval, Ph.D.
>>> ViQi Inc
>>> (805)-699-6081
>>> ___
>>> lustre-discuss mailing list
>>> lustre-discuss@lists.lustre.org
>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>
>>
>
>-- 
>Kris Kvilekval, Ph.D.
>ViQi Inc
>(805)-699-6081

>___
>lustre-discuss mailing list
>lustre-discuss@lists.lustre.org
>http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] reflecting state of underlying store in Lustre with HSM

[Kristian Kvilekval]

Thanks very much for the replies.. We are utilizing Lustre as a fast cache
on-top of S3.

AWS support has simply said they are not sure, but it is hard to break
through their defensive first line of support.
You have given me several avenues to explore. I did try the --exist flag,
but did not have a correct set of parameters.
I will also see if I can access the s3-copytool directly.
I will investigate and report back any findings.

Thank You,
Kris


On Wed, Jan 8, 2020 at 7:03 AM Degremont, Aurelien 
wrote:

> Hi Kris,
>
> As people said each HSM has its own copytool with its own constraints. To
> work properly with S3, additional metadata is stored in Lustre when files
> are imported from a S3 bucket.
> Questions specific to Amazon FSx For Lustre would be better asked to AWS
> support.
>
> Aurélien
>
> Le 08/01/2020 14:14, « Matt Rásó-Barnett »  a
> écrit :
>
> Hi Kris,
> I assume you are using Amazon's FSx for Lustre product
> (https://aws.amazon.com/fsx/lustre/) for this, rather than rolling
> Lustre HSM to S3 on AWS yourself?
>
> I'm afraid I don't know any more than you do on this product but it's
> something I've been keen to play with as it sounds really interesting.
>
> However the process or policy-engine by which Amazon are mapping S3
> objects to files in Lustre is not part of Lustre itself so you'd need
> input from Amazon on this - maybe Aurelien (in BCC) can comment here?
>
>  From their overview documenation my understanding is that the Lustre
> FS
> is meant to be short lived in this model, so you create it populated
> with the contents of the bucket *at that time*, run your workload,
> then
> archive results back and delete the FS.
>
> So if your bucket has been updated post Lustre FS creation, you may
> have
> to destroy the Lustre FS and recreate it to notice the changes. HSM
> isn't designed to be a mechanism for synchronising two endpoints that
> change independent of the other, so any change to the backend not via
> the filesystem will not be tracked.
>
> This is just me guessing without having used it however, I'll be
> interested to hear if you learn more about this from Amazon.
>
> Kind regards,
>
> Matt
>
> On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote:
> >We have Lustre <- HSM -> S3
> >
> >We have direct modifications to S3 that occur after the Lustre
> filesystem
> >is created
> >I was  wondering if there is any way to register a new/deleted file
> at the
> >Lustre level using HSM or other commands
> >
> >Say a user uploads a file to S3, and I know  the mapped path in
> Lustre,
> >I would like to do
> >lfs hsm_register /path/to/file/in/S3/ # Create a metadata entry in
> >Lustre
> >lfs hsm_restore /path/to/file/in/S3   # Fetch file from S3 into
> Lustre
> >
> >Thx
> >
> >
> >
> >
> >
> >
> >On Tue, Jan 7, 2020 at 8:04 AM Colin Faber  wrote:
> >
> >> Can you provide an example of what you're attempting to
> accomplish?  Am I
> >> understanding correctly, that you've got a lustre file system,
> you're then
> >> writing data into this file system?
> >>
> >> On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval 
> wrote:
> >>
> >>> We are using Lustre on AWS backed by S3 buckets.
> >>> When creating a new Lustre filesystem, S3 metadata can be
> automatically
> >>> imported  into Lustre.  When changes occur to the underlying S3
> store,
> >>> these changes are not automatically reflected.
> >>>
> >>> Is it possible to indicate the creation / deletion of the
> underlying S3
> >>> files after filesystem creation using HSM?
> >>> Is it possible to reimport the underlying metadata after creation?
> >>>
> >>> Any pointers appreciated.
> >>>
> >>> Thanks,
> >>> Kris
> >>>
> >>> --
> >>> Kris Kvilekval, Ph.D.
> >>> ViQi Inc
> >>> (805)-699-6081
> >>> ___
> >>> lustre-discuss mailing list
> >>> lustre-discuss@lists.lustre.org
> >>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> >>>
> >>
> >
> >--
> >Kris Kvilekval, Ph.D.
> >ViQi Inc
> >(805)-699-6081
>
> >___
> >lustre-discuss mailing list
> >lustre-discuss@lists.lustre.org
> >http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>


-- 
Kris Kvilekval, Ph.D.
ViQi Inc
(805)-699-6081
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Lemur Lustre - make rpm fails

[Pinkesh Valdria]

Hello Nathaniel, 

As a workaround,  is there an older lemur rpm version or older Lustre version I 
should use to unblock myself? 
https://github.com/whamcloud/lemur/issues/7

https://github.com/whamcloud/lemur/issues/8

Thanks,
Pinkesh Valdria


On 12/11/19, 6:31 AM, "Pinkesh Valdria"  wrote:

Hi Nathaniel, 

I have an issue ticket opened:  https://github.com/whamcloud/lemur/issues/7

I tried to do it locally,  that also fails, given below is the error.  

[root@lustre-client-4 lemur]# lfs --version
lfs 2.12.3
[root@lustre-client-4 lemur]# uname -a
Linux lustre-client-4 3.10.0-1062.7.1.el7.x86_64 #1 SMP Mon Dec 2 17:33:29 
UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
 [root@lustre-client-4 lemur]# lsb_release -r
Release:7.6.1810
[root@lustre-client-4 lemur]#


[root@lustre-client-4 lemur]# make local-rpm
make -C packaging/rpm NAME=lemur VERSION=0.6.0_4_g4655df8 RELEASE=1 
URL="https://github.com/intel-hpdd/lemur";
make[1]: Entering directory `/root/lemur/packaging/rpm'
cd ../../ && \
.



github.com/intel-hpdd/lemur/vendor/github.com/aws/aws-sdk-go/service/s3/s3manager
github.com/intel-hpdd/lemur/cmd/lhsm-plugin-s3
install -d $(dirname 
/root/rpmbuild/BUILDROOT/lemur-hsm-agent-0.6.0_4_g4655df8-1.x86_64//usr/bin/lhsm-plugin-s3)
install -m 755 lhsm-plugin-s3 
/root/rpmbuild/BUILDROOT/lemur-hsm-agent-0.6.0_4_g4655df8-1.x86_64//usr/bin/lhsm-plugin-s3
go build -v -i -ldflags "-X 'main.version=0.6.0_4_g4655df8'" -o lhsm 
./cmd/lhsm
github.com/intel-hpdd/lemur/vendor/github.com/intel-hpdd/go-lustre/pkg/pool
github.com/intel-hpdd/lemur/cmd/lhsmd/agent/fileid
github.com/intel-hpdd/lemur/vendor/github.com/intel-hpdd/go-lustre/llapi
github.com/intel-hpdd/lemur/vendor/gopkg.in/yaml.v2
github.com/intel-hpdd/lemur/vendor/gopkg.in/urfave/cli.v1
# github.com/intel-hpdd/lemur/vendor/github.com/intel-hpdd/go-lustre/llapi
cgo-gcc-prolog: In function '_cgo_c110903d49cd_C2func_llapi_get_version':
cgo-gcc-prolog:58:2: warning: 'llapi_get_version' is deprecated (declared 
at /usr/include/lustre/lustreapi.h:398) [-Wdeprecated-declarations]
cgo-gcc-prolog: In function '_cgo_c110903d49cd_Cfunc_llapi_get_version':
cgo-gcc-prolog:107:2: warning: 'llapi_get_version' is deprecated (declared 
at /usr/include/lustre/lustreapi.h:398) [-Wdeprecated-declarations]
# github.com/intel-hpdd/lemur/vendor/github.com/intel-hpdd/go-lustre/llapi
vendor/github.com/intel-hpdd/go-lustre/llapi/changelog.go:273:39: cannot 
use _Ctype_int(r.flags) (type _Ctype_int) as type int32 in argument to 
_Cfunc_hsm_get_cl_flags
make[2]: *** [lhsm] Error 2
make[2]: Leaving directory 
`/root/rpmbuild/BUILD/lemur-0.6.0_4_g4655df8/src/github.com/intel-hpdd/lemur'
error: Bad exit status from /var/tmp/rpm-tmp.cPPeEL (%install)


RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.cPPeEL (%install)
make[1]: *** [rpm] Error 1
make[1]: Leaving directory `/root/lemur/packaging/rpm'
make: *** [local-rpm] Error 2
[root@lustre-client-4 lemur]#



Thanks,
Pinkesh Valdria 



On 12/10/19, 4:55 AM, "lustre-discuss on behalf of Nathaniel Clark" 
 
wrote:

Can you open at ticket for this on 


https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_whamcloud_lemur_issues&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=HpfvG0tozSl7HgJJuyxxo2149EjwqpQDE7ytv-4sZuI&m=dvUy7ZhvTpzQ9yJzUhQmk0UHrXXOGiSc2X1_Sm5yOhY&s=aD2CBP6CmEF14pb7PM2A-H4aFyzbd09y5IRcQXqIHj8&e=
 

And possibly


https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.whamcloud.com_projects_LMR&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=HpfvG0tozSl7HgJJuyxxo2149EjwqpQDE7ytv-4sZuI&m=dvUy7ZhvTpzQ9yJzUhQmk0UHrXXOGiSc2X1_Sm5yOhY&s=SoLFFKtz2XY9CNh4vyFssmTyhvmkIqyABrH_FzZzUQk&e=
 



You could also try:

$ make local-rpm



Which will avoid the docker stack and just build on the local machine

(beware it sudo's to install rpm build dependencies).





-- 

Nathaniel Clark 

Senior Engineer

Whamcloud / DDN



On Mon, 2019-12-09 at 15:04 -0800, Pinkesh Valdria wrote:

> I am trying to install Lemur on CentOS 7.6 (7.6.1810) to integrate

> with Object storage but the install fails.   I used the instructions

> on below page to install.  I already had Lustre client (2.12.3)

> installed on the machine,  so I started with steps for Lemur.

>  

> 
https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.whamcloud.com_display_PUB_HPDD-2BHSM-2BAgent-2Band-2BData-2B

Re: [lustre-discuss] reflecting state of underlying store in Lustre with HSM

[Andreas Dilger]

On Jan 8, 2020, at 06:13, Matt Rásó-Barnett 
mailto:m...@rasobarnett.com>> wrote:

Hi Kris,
I assume you are using Amazon's FSx for Lustre product 
(https://aws.amazon.com/fsx/lustre/) for this, rather than rolling Lustre HSM 
to S3 on AWS yourself?

I'm afraid I don't know any more than you do on this product but it's something 
I've been keen to play with as it sounds really interesting.

However the process or policy-engine by which Amazon are mapping S3 objects to 
files in Lustre is not part of Lustre itself so you'd need input from Amazon on 
this - maybe Aurelien (in BCC) can comment here?

From their overview documenation my understanding is that the Lustre FS is 
meant to be short lived in this model, so you create it populated with the 
contents of the bucket *at that time*, run your workload, then archive results 
back and delete the FS.

So if your bucket has been updated post Lustre FS creation, you may have to 
destroy the Lustre FS and recreate it to notice the changes. HSM isn't designed 
to be a mechanism for synchronising two endpoints that change independent of 
the other, so any change to the backend not via the filesystem will not be 
tracked.

Wouldn't it be enough to "lfs hsm_release  ..." from Lustre, so that 
Lustre drops the local copy of the file, and it is imported again from S3?  
There would need to be some mechanism to run this command when the S3 object is 
modified (preferably automatically), which is outside the scope of Lustre.  Are 
there S3 events that could be hooked in this case to trigger an action, or a 
Changelog equivalent?

Cheers, Andreas

On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote:
We have Lustre <- HSM -> S3

We have direct modifications to S3 that occur after the Lustre filesystem
is created
I was  wondering if there is any way to register a new/deleted file at the
Lustre level using HSM or other commands

Say a user uploads a file to S3, and I know  the mapped path in Lustre,
I would like to do
lfs hsm_register /path/to/file/in/S3/ # Create a metadata entry in
Lustre
lfs hsm_restore /path/to/file/in/S3   # Fetch file from S3 into Lustre

Thx






On Tue, Jan 7, 2020 at 8:04 AM Colin Faber 
mailto:cfa...@gmail.com>> wrote:

Can you provide an example of what you're attempting to accomplish?  Am I
understanding correctly, that you've got a lustre file system, you're then
writing data into this file system?

On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval 
mailto:k...@viqi.org>> wrote:

We are using Lustre on AWS backed by S3 buckets.
When creating a new Lustre filesystem, S3 metadata can be automatically
imported  into Lustre.  When changes occur to the underlying S3 store,
these changes are not automatically reflected.

Is it possible to indicate the creation / deletion of the underlying S3
files after filesystem creation using HSM?
Is it possible to reimport the underlying metadata after creation?

Any pointers appreciated.

Thanks,
Kris

--
Kris Kvilekval, Ph.D.
ViQi Inc
(805)-699-6081
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org



--
Kris Kvilekval, Ph.D.
ViQi Inc
(805)-699-6081

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Cheers, Andreas
--
Andreas Dilger
Principal Lustre Architect
Whamcloud






___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] reflecting state of underlying store in Lustre with HSM

[Kristian Kvilekval]

Hi Andreas,

 AWS Lustre takes a snapshot of S3 file metadata on creation.
 However, I am trying to include files that are added  to the S3 bucket
after Lustre filesystem creation.   Since I know which  files were added, I
wanted to simply inform the Lustre HSM layer of their existence.   At this
point I could unmount and re-create  the filesystem, or I must copy the
file from s3 to  Lustre  directly bypassing the HSM.  Both of these options
are pretty bad IMHO, since the HSM is more efficient and has better
performance.

Kris


On Wed, Jan 8, 2020 at 3:15 PM Andreas Dilger  wrote:

> On Jan 8, 2020, at 06:13, Matt Rásó-Barnett  wrote:
>
>
> Hi Kris,
> I assume you are using Amazon's FSx for Lustre product (
> https://aws.amazon.com/fsx/lustre/) for this, rather than rolling Lustre
> HSM to S3 on AWS yourself?
>
> I'm afraid I don't know any more than you do on this product but it's
> something I've been keen to play with as it sounds really interesting.
>
> However the process or policy-engine by which Amazon are mapping S3
> objects to files in Lustre is not part of Lustre itself so you'd need input
> from Amazon on this - maybe Aurelien (in BCC) can comment here?
>
> From their overview documenation my understanding is that the Lustre FS is
> meant to be short lived in this model, so you create it populated with the
> contents of the bucket *at that time*, run your workload, then archive
> results back and delete the FS.
>
> So if your bucket has been updated post Lustre FS creation, you may have
> to destroy the Lustre FS and recreate it to notice the changes. HSM isn't
> designed to be a mechanism for synchronising two endpoints that change
> independent of the other, so any change to the backend not via the
> filesystem will not be tracked.
>
>
> Wouldn't it be enough to "lfs hsm_release  ..." from Lustre, so that
> Lustre drops the local copy of the file, and it is imported again from S3?
> There would need to be some mechanism to run this command when the S3
> object is modified (preferably automatically), which is outside the scope
> of Lustre.  Are there S3 events that could be hooked in this case to
> trigger an action, or a Changelog equivalent?
>
> Cheers, Andreas
>
> On Tue, Jan 07, 2020 at 03:18:39PM -0800, Kristian Kvilekval wrote:
>
> We have Lustre <- HSM -> S3
>
> We have direct modifications to S3 that occur after the Lustre filesystem
> is created
> I was  wondering if there is any way to register a new/deleted file at the
> Lustre level using HSM or other commands
>
> Say a user uploads a file to S3, and I know  the mapped path in Lustre,
> I would like to do
> lfs hsm_register /path/to/file/in/S3/ # Create a metadata entry in
> Lustre
> lfs hsm_restore /path/to/file/in/S3   # Fetch file from S3 into Lustre
>
> Thx
>
>
>
>
>
>
> On Tue, Jan 7, 2020 at 8:04 AM Colin Faber  wrote:
>
> Can you provide an example of what you're attempting to accomplish?  Am I
> understanding correctly, that you've got a lustre file system, you're then
> writing data into this file system?
>
> On Mon, Jan 6, 2020 at 10:02 PM Kristian Kvilekval  wrote:
>
> We are using Lustre on AWS backed by S3 buckets.
> When creating a new Lustre filesystem, S3 metadata can be automatically
> imported  into Lustre.  When changes occur to the underlying S3 store,
> these changes are not automatically reflected.
>
> Is it possible to indicate the creation / deletion of the underlying S3
> files after filesystem creation using HSM?
> Is it possible to reimport the underlying metadata after creation?
>
> Any pointers appreciated.
>
> Thanks,
> Kris
>
> --
> Kris Kvilekval, Ph.D.
> ViQi Inc
> (805)-699-6081
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
>
> --
> Kris Kvilekval, Ph.D.
> ViQi Inc
> (805)-699-6081
>
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
> Cheers, Andreas
> --
> Andreas Dilger
> Principal Lustre Architect
> Whamcloud
>
>
>
>
>
>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>


-- 
Kris Kvilekval, Ph.D.
ViQi Inc
(805)-699-6081
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org