Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-21 Thread Anshuman Aggarwal
I'd a appreciate any help/pointers in implementing the proposal below including the right path to get this into the kernel itself. -- I'm outlining below a proposal for a RAID device mapper virtual block device for the kernel which adds "split raid" functionality on

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-21 Thread Greg Freemyer
On November 21, 2014 5:15:43 AM EST, Anshuman Aggarwal wrote: >I'd a appreciate any help/pointers in implementing the proposal below >including the right path to get this into the kernel itself. >-- >I'm outlining below a proposal for a RAID device mapper virtual

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-21 Thread Anshuman Aggarwal
N pass through but with their own filesystems. Concatenation is via some kind of union fs solution not at the block level. Data is not supposed to be striped (this is critical so as to prevent all drives to be required to be accessed for consecutive data) Idea is that each drive can work independe

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-22 Thread Greg Freemyer
Top posting is strongly discouraged on all kernel related mailing lists including this one. I've moved your reply to the bottom and then replied after that. In future I will ignore replies that are top posted. >On 21 November 2014 17:11, Greg Freemyer >wrote: >> >> >> On November 21, 2014 5:

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-22 Thread Anshuman Aggarwal
On 22 November 2014 at 18:47, Greg Freemyer wrote: > Top posting is strongly discouraged on all kernel related mailing lists > including this one. I've moved your reply to the bottom and then replied > after that. In future I will ignore replies that are top posted. > > >>On 21 November 2014 1

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-22 Thread Greg Freemyer
On Sat, Nov 22, 2014 at 8:22 AM, Anshuman Aggarwal wrote: > By not using stripes, we restrict writes to happen to just 1 drive and > the XOR output to the parity drive which then explains the delayed and > batched checksum (resulting in fewer writes to the parity drive). The > intention is that if

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-22 Thread Anshuman Aggarwal
On 22 November 2014 at 19:33, Greg Freemyer wrote: > On Sat, Nov 22, 2014 at 8:22 AM, Anshuman Aggarwal > wrote: >> By not using stripes, we restrict writes to happen to just 1 drive and >> the XOR output to the parity drive which then explains the delayed and >> batched checksum (resulting in fe

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-22 Thread Greg Freemyer
On November 22, 2014 9:43:23 AM EST, Anshuman Aggarwal wrote: >On 22 November 2014 at 19:33, Greg Freemyer >wrote: >> On Sat, Nov 22, 2014 at 8:22 AM, Anshuman Aggarwal >> wrote: >>> By not using stripes, we restrict writes to happen to just 1 drive >and >>> the XOR output to the parity drive

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-23 Thread SandeepKsinha
On Sat, Nov 22, 2014 at 8:24 PM, Greg Freemyer wrote: > > > On November 22, 2014 9:43:23 AM EST, Anshuman Aggarwal < > anshuman.aggar...@gmail.com> wrote: > >On 22 November 2014 at 19:33, Greg Freemyer > >wrote: > >> On Sat, Nov 22, 2014 at 8:22 AM, Anshuman Aggarwal > >> wrote: > >>> By not us

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-23 Thread Anshuman Aggarwal
Sandeep, This isn't exactly RAID4 (only thing in common is a single parity disk but the data is not striped at all). I did bring it up on the linux-raid mailing list and have had a short conversation with Neil. He wasn't too excited about device mapper but didn't indicate why or why not. I would

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-24 Thread Greg Freemyer
On November 24, 2014 1:48:48 AM EST, Anshuman Aggarwal wrote: >Sandeep, > This isn't exactly RAID4 (only thing in common is a single parity >disk but the data is not striped at all). I did bring it up on the >linux-raid mailing list and have had a short conversation with Neil. >He wasn't too ex

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-24 Thread Anshuman Aggarwal
On 24 November 2014 at 18:49, Greg Freemyer wrote: > > > On November 24, 2014 1:48:48 AM EST, Anshuman Aggarwal > wrote: >>Sandeep, >> This isn't exactly RAID4 (only thing in common is a single parity >>disk but the data is not striped at all). I did bring it up on the >>linux-raid mailing list

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-24 Thread Valdis . Kletnieks
On Mon, 24 Nov 2014 22:58:08 +0530, Anshuman Aggarwal said: > prevents it from directly recognized by file system code . I was > wondering if Split RAID block devices can be made to be unaware to the > RAID scheme on top and be fully mountable and usable without the raid > drivers (of course inval

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-24 Thread Greg Freemyer
On November 24, 2014 12:28:08 PM EST, Anshuman Aggarwal wrote: >On 24 November 2014 at 18:49, Greg Freemyer >wrote: >> >> >> On November 24, 2014 1:48:48 AM EST, Anshuman Aggarwal > wrote: >>>Sandeep, >>> This isn't exactly RAID4 (only thing in common is a single parity >>>disk but the data is

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-29 Thread Anshuman Aggarwal
On 25 November 2014 at 10:26, Greg Freemyer wrote: > > > On November 24, 2014 12:28:08 PM EST, Anshuman Aggarwal > wrote: >>On 24 November 2014 at 18:49, Greg Freemyer >>wrote: >>> >>> >>> On November 24, 2014 1:48:48 AM EST, Anshuman Aggarwal >> wrote: Sandeep, This isn't exactly RAID

Re: Split RAID: Proposal for archival RAID using incremental batch checksum

2014-11-29 Thread Greg Freemyer
On Thu, Nov 27, 2014 at 12:50 PM, Anshuman Aggarwal wrote: > On 25 November 2014 at 10:26, Greg Freemyer wrote: >> >> >> On November 24, 2014 12:28:08 PM EST, Anshuman Aggarwal >> wrote: >>>On 24 November 2014 at 18:49, Greg Freemyer >>>wrote: > Also if I don't store the metadata on