Re: [gpfsug-discuss] GPFS de duplication
On 20/05/2021 13:58, Dave Bond wrote: As part of a project I am doing I am looking if there are any de duplication options for GPFS? I see there is no native de dupe for the filesystem. The scenario would be user A creates a file or folder and user B takes a copy within the same filesystem, though separate independent filesets. The intention would be to store 1 copy. So I was wondering 1) Is this is planned to be implemented into GPFS in the future? 2) Is anyone is using any other solutions that have good GPFS integration? Disk space in 2021 is insanely cheap. With an ESS/DSS you can get many PB in a single rack. The complexity that dedup introduces is simple not worth it IMHO. Or put another way there is better things the developers at IBM can be working on than dedup code. Historically if you crunched the numbers the licensing for dedup on NetApp was similar to just buying more disk unless you where storing hundreds of copies of the same data. About the only use case scenario would be storing lots of virtual machines. However I refer you back to my original point :-) JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] GPFS de duplication
Do file clones meet the workflow requirement? That is, can you control from whence the second (and further) copies are made? -- Stephen > On May 20, 2021, at 9:01 AM, Andrew Beattie wrote: > > Dave, > > Spectrum Scale does not support de-duplication, it does support compression. > You can however use block storage that supports over subscription / thin > provisioning / deduplication for data only NSD’s, we do not recommend them > for metadata. > > In your scenario is user B planning on making changes to the data which is > why you need a copy? > > I know of customers that do this regularly with block storage such as the IBM > Flashsystem product family In conjunction with IBM Spectrum Copy Data > Management. But I don’t believe CDM supports file based storage. > > Regards, > > Andrew Beattie > Technical Sales Specialist > Storage for Data and AI > IBM Australia and New Zealand > P. +61 421 337 927 > E. abeat...@au1.ibm.com > >>> On 20 May 2021, at 22:58, Dave Bond wrote: >>> >> >> >> >> Hello >> >> As part of a project I am doing I am looking if there are any de duplication >> options for GPFS? I see there is no native de dupe for the filesystem. The >> scenario would be user A creates a file or folder and user B takes a copy >> within the same filesystem, though separate independent filesets. The >> intention would be to store 1 copy.So I was wondering >> >> 1) Is this is planned to be implemented into GPFS in the future? >> 2) Is anyone is using any other solutions that have good GPFS integration? >> >> Dave > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] GPFS de duplication
Dave, Spectrum Scale does not support de-duplication, it does support compression. You can however use block storage that supports over subscription / thin provisioning / deduplication for data only NSD’s, we do not recommend them for metadata. In your scenario is user B planning on making changes to the data which is why you need a copy? I know of customers that do this regularly with block storage such as the IBM Flashsystem product family In conjunction with IBM Spectrum Copy Data Management. But I don’t believe CDM supports file based storage. Regards, Andrew Beattie Technical Sales Specialist Storage for Data and AI IBM Australia and New Zealand P. +61 421 337 927 E. abeat...@au1.ibm.com > On 20 May 2021, at 22:58, Dave Bond wrote: > > > This Message Is From an External Sender > This message came from outside your organization. > > > Hello > > As part of a project I am doing I am looking if there are any de duplication > options for GPFS? I see there is no native de dupe for the filesystem. The > scenario would be user A creates a file or folder and user B takes a copy > within the same filesystem, though separate independent filesets. The > intention would be to store 1 copy.So I was wondering > > 1) Is this is planned to be implemented into GPFS in the future? > 2) Is anyone is using any other solutions that have good GPFS integration? > > Dave ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
