Re: HDD long-term data storage with ensured integrity

2024-05-04 Thread Marc SCHAEFER
On Fri, May 03, 2024 at 01:50:52PM -0700, David Christensen wrote: > Thank you for devising a benchmark and posting some data. :-) I did not do the comparison hosted on github. I just wrote the script which tests the dm-integrity on dm-raid error detection and error correction. > FreeBSD also

Re: HDD long-term data storage with ensured integrity

2024-05-03 Thread David Christensen
On 5/3/24 04:26, Marc SCHAEFER wrote: On Mon, Apr 08, 2024 at 10:04:01PM +0200, Marc SCHAEFER wrote: For off-site long-term offline archiving, no, I am not using RAID. Now, as I had to think a bit about ONLINE integrity, I found this comparison:

Re: HDD long-term data storage with ensured integrity

2024-05-03 Thread Michael Kjörling
On 3 May 2024 13:26 +0200, from schae...@alphanet.ch (Marc SCHAEFER): > https://github.com/t13a/dm-integrity-benchmarks > > Contenders are btrfs, zfs, and notably ext4+dm-integrity+dm-raid ZFS' selling point is not performance, _especially_ on rotational drives. In fact, it's fairly widely

Re: HDD long-term data storage with ensured integrity

2024-05-03 Thread Marc SCHAEFER
On Mon, Apr 08, 2024 at 10:04:01PM +0200, Marc SCHAEFER wrote: > For off-site long-term offline archiving, no, I am not using RAID. Now, as I had to think a bit about ONLINE integrity, I found this comparison: https://github.com/t13a/dm-integrity-benchmarks Contenders are btrfs, zfs, and

Re: HDD long-term data storage with ensured integrity

2024-04-12 Thread David Christensen
On 4/12/24 08:14, piorunz wrote: On 10/04/2024 12:10, David Christensen wrote: Those sound like some compelling features. I believe the last time I tried Btrfs was Debian 9 (?).  I ran into problems because I did not do the required manual maintenance (rebalancing).  Does the Btrfs in Debian

Re: HDD long-term data storage with ensured integrity

2024-04-12 Thread piorunz
On 10/04/2024 12:10, David Christensen wrote: Those sound like some compelling features. I believe the last time I tried Btrfs was Debian 9 (?).  I ran into problems because I did not do the required manual maintenance (rebalancing).  Does the Btrfs in Debian 11 or Debian 12 still require

Re: HDD long-term data storage with ensured integrity

2024-04-10 Thread David Christensen
On 4/10/24 08:49, Paul Leiber wrote: Am 10.04.2024 um 13:10 schrieb David Christensen: Does the Btrfs in Debian 11 or Debian 12 still require manual maintenance?  If so, what and how often? Scrub and balance are actions which have been recommended. I am using btrfsmaintenance scripts [1][2]

Re: HDD long-term data storage with ensured integrity

2024-04-10 Thread Paul Leiber
Am 10.04.2024 um 13:10 schrieb David Christensen: On 4/9/24 17:08, piorunz wrote: On 02/04/2024 13:53, David Christensen wrote: Does anyone have any comments or suggestions regarding how to use magnetic hard disk drives, commodity x86 computers, and Debian for long-term data storage with

Re: HDD long-term data storage with ensured integrity

2024-04-10 Thread Curt
On 2024-04-10, David Christensen wrote: >> >> I use Btrfs, on all my systems, including some servers, with soft Raid1 >> and Raid10 modes (because these modes are considered stable and >> production ready). I decided on Btrfs not ZFS, because Btrfs allows to >> migrate drives on the fly while

Re: HDD long-term data storage with ensured integrity

2024-04-10 Thread David Christensen
On 4/9/24 17:08, piorunz wrote: On 02/04/2024 13:53, David Christensen wrote: Does anyone have any comments or suggestions regarding how to use magnetic hard disk drives, commodity x86 computers, and Debian for long-term data storage with ensured integrity? I use Btrfs, on all my systems,

Re: HDD long-term data storage with ensured integrity

2024-04-09 Thread piorunz
On 02/04/2024 13:53, David Christensen wrote: Does anyone have any comments or suggestions regarding how to use magnetic hard disk drives, commodity x86 computers, and Debian for long-term data storage with ensured integrity? I use Btrfs, on all my systems, including some servers, with soft

Re: HDD long-term data storage with ensured integrity

2024-04-08 Thread David Christensen
On 4/8/24 13:04, Marc SCHAEFER wrote: Hello, On Mon, Apr 08, 2024 at 11:28:04AM -0700, David Christensen wrote: So, an ext4 file system on an LVM logical volume? Why LVM? Are you implementing redundancy (RAID)? Is your data larger than a single disk (concatenation/ JBOD)? Something else?

Re: HDD long-term data storage with ensured integrity

2024-04-08 Thread Marc SCHAEFER
Hello, On Mon, Apr 08, 2024 at 11:28:04AM -0700, David Christensen wrote: > So, an ext4 file system on an LVM logical volume? > > Why LVM? Are you implementing redundancy (RAID)? Is your data larger than > a single disk (concatenation/ JBOD)? Something else? For off-site long-term offline

Re: HDD long-term data storage with ensured integrity

2024-04-08 Thread David Christensen
On 4/8/24 02:38, Marc SCHAEFER wrote: For offline storage: On Tue, Apr 02, 2024 at 05:53:15AM -0700, David Christensen wrote: Does anyone have any comments or suggestions regarding how to use magnetic hard disk drives, commodity x86 computers, and Debian for long-term data storage with ensured

Re: HDD long-term data storage with ensured integrity

2024-04-08 Thread Marc SCHAEFER
For offline storage: On Tue, Apr 02, 2024 at 05:53:15AM -0700, David Christensen wrote: > Does anyone have any comments or suggestions regarding how to use magnetic > hard disk drives, commodity x86 computers, and Debian for long-term data > storage with ensured integrity? I use LVM on ext4, and

Re: HDD long-term data storage with ensured integrity

2024-04-03 Thread Jonathan Dowland
On Tue Apr 2, 2024 at 10:57 PM BST, David Christensen wrote: > AIUI neither LVM nor ext4 have data and metadata checksum and correction > features. But, it should be possible to achieve such by including > dm-integrity (for checksumming) and some form of RAID (for correction) > in the storage

Re: HDD long-term data storage with ensured integrity

2024-04-03 Thread David Christensen
On 4/2/24 14:57, David Christensen wrote: AIUI neither LVM nor ext4 have data and metadata checksum and correction features.  But, it should be possible to achieve such by including dm-integrity (for checksumming) and some form of RAID (for correction) in the storage stack.  I need to explore

Re: HDD long-term data storage with ensured integrity

2024-04-02 Thread David Christensen
On 4/2/24 06:55, Stefan Monnier wrote: The most obvious alternative to ZFS on Debian would be Btrfs. Does anyone have any comments or suggestions regarding Btrfs and data corruption bugs, concurrency, CMM level, PSP, etc.? If you're worried about such things, I'd think "the most obvious

Re: HDD long-term data storage with ensured integrity

2024-04-02 Thread Stefan Monnier
> The most obvious alternative to ZFS on Debian would be Btrfs. Does anyone > have any comments or suggestions regarding Btrfs and data corruption bugs, > concurrency, CMM level, PSP, etc.? If you're worried about such things, I'd think "the most obvious alternative" is LVM+ext4. Both Btrfs and