[ceph-users] Re: How to solve data fixity

2024-02-09 Thread Casey Bodley
i've cc'ed Matt who's working on the s3 object integrity feature https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html, where rgw compares the generated checksum with the client's on ingest, then stores it with the object so clients can read it back for later integrit

[ceph-users] Re: How to solve data fixity

2024-02-09 Thread Josh Baergen
MPU etags are an MD5-of-MD5s, FWIW. If the users knows how the parts are uploaded then it can be used to verify contents, both just after upload and then at download time (both need to be validated if you want end-to-end validation - but then you're trusting the system to not change the etag undern

[ceph-users] Re: How to solve data fixity

2024-02-09 Thread Anthony D'Atri
> On Feb 9, 2024, at 08:15, Michal Strnad wrote: > > Thank you for your response. > > We have already done some Lua scripting in the past, and it wasn't entirely > enjoyable :-), but we may have to do it again. Scrubbing is still enabled, > and turning it off definitely won't be an option.

[ceph-users] Re: How to solve data fixity

2024-02-09 Thread Michal Strnad
Thank you for your response. We have already done some Lua scripting in the past, and it wasn't entirely enjoyable :-), but we may have to do it again. Scrubbing is still enabled, and turning it off definitely won't be an option. However, due to the project requirements, it would be great if

[ceph-users] Re: How to solve data fixity

2024-02-09 Thread Anthony D'Atri
You could use Lua scripting perhaps to do this at ingest, but I'm very curious about scrubs -- you have them turned off completely? > On Feb 9, 2024, at 04:18, Michal Strnad wrote: > > Hi all! > > In the context of a repository-type project, we need to address a situation > where we cannot u