Re: [lustre-discuss] status of HSM copytools?
Hi The Compute Canada S3/Ceph copytool was never deployed in production since S3 backed storage was never installed. We moved to a tape based system with TSM instead and have been running that copytool on 3 different sites for the past year. https://github.com/guilbaults/ct_tsm On Sun, Aug 23, 2020 at 9:34 AM Matt Rásó-Barnett wrote: > Hi Andrew, > > We are using Lemur in production at Cambridge with our Lustre HSM setup, > although in our case we are using the POSIX backend of it rather than > the S3 backend. > > We adopted it early at the time it was being developed under Intel, as > the prospect of commercial support (we were Intel IEEL customers at the > time), and some of the promising features that were being worked on with > it (snapshot versions, checksums, performance), was very enticing. > > But as you've seen, it was a casualty of Intel dropping Lustre, so it's > not being developed any more. We unfortunately don't have the skills or > resources internally to develop it ourselves, so I personally am looking > at how we will transition away from it in the coming months as we > refresh our HSM setup. I may switch back to using the robinhood > developed lhsmtool_cmd with our POSIX backends for example. > > Lemur is largely doing the job for us however, so it reached a decent > stage of development before Intel dropped it, but I still wouldn't > recommend starting with it without someone backing maintenance of it. > > From speaking with the Amazon representative at last year's LUG, I > understood that they are using Lemur internally for their HSM > integration as well. I had meant to follow up with them to see if they > would consider open-sourcing their fork of the project as to my > knowledge there aren't any other major users of Lemur. > > I'd also be interested as you are in hearing what others are using, > commericial or otherwise. I've never had the feeling that HSM is > particularly widely used so it'd be good to share information for those > of us who are! > > Best wishes, > > Matt > > On Sat, Aug 22, 2020 at 10:54:23PM +1000, Andrew Elwell wrote: > >Hi folks, > >I'm looking round to see what's current / 'supported' / working in the > >state of copytools. Ideally one that can migrate to/from object stores > >(Ceph or S3). The github repo for Lemur > >(https://github.com/whamcloud/lemur/commits/master) doesn't seem to > >have had any substantial work since it left Intel - unlucky timing > >with the owner shift? > >I've seen another from Compute Canada > >(https://github.com/ComputeCanada/lustre-obj-copytool) but that too > >hasn't been touched for years. > > > >Anyone care to comment on some working ones? Horror stories? Ones to > avoid? > >hey, I'm even (I'll probably regret this) open to _email_ from > >salesdroids if you have a working product and can point me to some > >users (but don't try and phone me or make me sit through a webinar). > > > >Many thanks > > > > > >Andrew > >___ > >lustre-discuss mailing list > >lustre-discuss@lists.lustre.org > >http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] status of HSM copytools?
Hi Andrew, We are using Lemur in production at Cambridge with our Lustre HSM setup, although in our case we are using the POSIX backend of it rather than the S3 backend. We adopted it early at the time it was being developed under Intel, as the prospect of commercial support (we were Intel IEEL customers at the time), and some of the promising features that were being worked on with it (snapshot versions, checksums, performance), was very enticing. But as you've seen, it was a casualty of Intel dropping Lustre, so it's not being developed any more. We unfortunately don't have the skills or resources internally to develop it ourselves, so I personally am looking at how we will transition away from it in the coming months as we refresh our HSM setup. I may switch back to using the robinhood developed lhsmtool_cmd with our POSIX backends for example. Lemur is largely doing the job for us however, so it reached a decent stage of development before Intel dropped it, but I still wouldn't recommend starting with it without someone backing maintenance of it. From speaking with the Amazon representative at last year's LUG, I understood that they are using Lemur internally for their HSM integration as well. I had meant to follow up with them to see if they would consider open-sourcing their fork of the project as to my knowledge there aren't any other major users of Lemur. I'd also be interested as you are in hearing what others are using, commericial or otherwise. I've never had the feeling that HSM is particularly widely used so it'd be good to share information for those of us who are! Best wishes, Matt On Sat, Aug 22, 2020 at 10:54:23PM +1000, Andrew Elwell wrote: Hi folks, I'm looking round to see what's current / 'supported' / working in the state of copytools. Ideally one that can migrate to/from object stores (Ceph or S3). The github repo for Lemur (https://github.com/whamcloud/lemur/commits/master) doesn't seem to have had any substantial work since it left Intel - unlucky timing with the owner shift? I've seen another from Compute Canada (https://github.com/ComputeCanada/lustre-obj-copytool) but that too hasn't been touched for years. Anyone care to comment on some working ones? Horror stories? Ones to avoid? hey, I'm even (I'll probably regret this) open to _email_ from salesdroids if you have a working product and can point me to some users (but don't try and phone me or make me sit through a webinar). Many thanks Andrew ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] status of HSM copytools?
Hi folks, I'm looking round to see what's current / 'supported' / working in the state of copytools. Ideally one that can migrate to/from object stores (Ceph or S3). The github repo for Lemur (https://github.com/whamcloud/lemur/commits/master) doesn't seem to have had any substantial work since it left Intel - unlucky timing with the owner shift? I've seen another from Compute Canada (https://github.com/ComputeCanada/lustre-obj-copytool) but that too hasn't been touched for years. Anyone care to comment on some working ones? Horror stories? Ones to avoid? hey, I'm even (I'll probably regret this) open to _email_ from salesdroids if you have a working product and can point me to some users (but don't try and phone me or make me sit through a webinar). Many thanks Andrew ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org