[PATCH v5 3/3] btrfs-progs: Adds new par3456 modes to support up to six parities

2014-02-24 Thread Andrea Mazzoleni
, and updates all the code to use them. Signed-off-by: Andrea Mazzoleni amadva...@gmail.com --- Makefile| 14 ++- chunk-recover.c | 18 +--- cmds-balance.c | 20 +++- cmds-check.c| 7 +- cmds-chunk.c| 18 +--- cmds-filesystem.c | 12 ++- ctree.h

[PATCH v5 0/3] New RAID library supporting up to six parities

2014-02-24 Thread Andrea Mazzoleni
(). - Converts the asm_begin/end() macros to inlined functions. - Fixes some more checkpatch.pl warnings. - Other minor style/comment changes. Andrea Mazzoleni (2): lib: raid: New RAID library supporting up to six parities fs: btrfs: Adds new par3456 modes to support up to six parities fs/btrfs

[PATCH v5 2/3] fs: btrfs: Adds new par3456 modes to support up to six parities

2014-02-24 Thread Andrea Mazzoleni
to six parities, and updates all the code to use them. Signed-off-by: Andrea Mazzoleni amadva...@gmail.com --- fs/btrfs/Kconfig | 1 + fs/btrfs/ctree.h | 50 ++-- fs/btrfs/disk-io.c | 7 +- fs/btrfs/extent-tree.c | 67 +++ fs/btrfs/inode.c

Triple parity and beyond

2013-11-18 Thread Andrea Mazzoleni
Hi, I want to report that I recently implemented a support for arbitrary number of parities that could be useful also for Linux RAID and Btrfs, both currently limited to double parity. In short, to generate the parity I use a Cauchy matrix specifically built to be compatible with the existing

Re: Triple parity and beyond

2013-11-18 Thread Andrea Mazzoleni
the Cauchy matrix with some explanation in the comments at (see the set_cauchy() function) : http://sourceforge.net/p/snapraid/code/ci/master/tree/mktables.c Ciao, Andrea On Mon, Nov 18, 2013 at 11:12 PM, H. Peter Anvin h...@zytor.com wrote: On 11/18/2013 02:08 PM, Andrea Mazzoleni wrote: Hi, I want

Re: Triple parity and beyond

2013-11-19 Thread Andrea Mazzoleni
for Extended Cauchy) Ciao, Andrea On Tue, Nov 19, 2013 at 12:25 AM, H. Peter Anvin h...@zytor.com wrote: On 11/18/2013 02:35 PM, Andrea Mazzoleni wrote: Hi Peter, The Cauchy matrix has the mathematical property to always have itself and all submatrices not singular. So, we are sure that we can

Re: Triple parity and beyond

2013-11-19 Thread Andrea Mazzoleni
: On 11/18/2013 02:35 PM, Andrea Mazzoleni wrote: Hi Peter, The Cauchy matrix has the mathematical property to always have itself and all submatrices not singular. So, we are sure that we can always solve the equations to recover the data disks. Besides the mathematical proof, I've also inverted

Re: Triple parity and beyond

2013-11-20 Thread Andrea Mazzoleni
Hi David, The choice of ZFS to use powers of 4 was likely not optimal, because to multiply by 4, it has to do two multiplications by 2. I can agree with that. I didn't copy ZFS's choice here David, it was not my intention to suggest that you copied from ZFS. Sorry to have expressed myself

Re: Triple parity and beyond

2013-11-20 Thread Andrea Mazzoleni
Hi John, Yes. There are still AMD CPUs sold without SSSE3. Most notably Athlon. Instead, Intel is providing SSSE3 from the Core 2 Duo. A detailed list is available at: http://en.wikipedia.org/wiki/SSSE3 Ciao, Andrea On Wed, Nov 20, 2013 at 7:09 PM, John Williams jwilliams4...@gmail.com wrote:

Re: Triple parity and beyond

2013-11-20 Thread Andrea Mazzoleni
%xmm5,%xmm2); where xmm2 is the intput/output Ciao, Andrea On Wed, Nov 20, 2013 at 7:43 PM, H. Peter Anvin h...@zytor.com wrote: It is also possible to quickly multiply by 2^-1 which makes for an interesting R parity. Andrea Mazzoleni amadva...@gmail.com wrote: Hi David, The choice of ZFS

Re: Triple parity and beyond

2013-11-20 Thread Andrea Mazzoleni
Hi Jim, I build the matrix in a way that results in coefficients matching Linux RAID for the first two rows, and at the same time gives the guarantee that all the square submatrices are not singular, resulting in a MDS code. I start forming a Cauchy matrix setting each element to 1/(xi+yj) where

Re: Triple parity and beyond

2013-11-20 Thread Andrea Mazzoleni
Hi Peter, static inline uint64_t d2_64(uint64_t v) { uint64_t mask = v 0x0101010101010101U; mask = (mask 8) - mask; (mask 7) I assume... No. It's (mask 8) - mask. We want to expand the bit at position 0 (in each byte) to the full byte, resulting in 0xFF if the bit is at

Re: Triple parity and beyond

2013-11-20 Thread Andrea Mazzoleni
Hi Peter, Now, that doesn't sound like something that can get neatly meshed into the Cauchy matrix scheme, I assume. You are correct. Multiplication by 2^-1 cannot be used for the Cauchy method. I used it to implement an alternate triple parity not requiring PSHUFB that I used as reference for

Re: Triple parity and beyond

2013-11-20 Thread Andrea Mazzoleni
Hi, First, create a 3 by 6 cauchy matrix, using x_i = 2^-i, and y_i = 0 for i=0, and y_i = 2^i for other i. In this case: x = { 1, 142, 71, 173, 216, 108 } y = { 0, 2, 4). The cauchy matrix is: 1 2 4 8 16 32 244 83 78 183 118 47 167 39 213 59 153 82 Divide row 2 by

Re: Triple parity and beyond

2013-11-20 Thread Andrea Mazzoleni
Hi Piergiorgio, In RAID-6 (as per raid6check) there is an easy way to verify where an HDD has incorrect data. I suspect, for each 2 parity block it should be possible to find 1 error (and if this is true, then quad parity is more attractive than triple one). Yes. The theory say that with

Re: Triple parity and beyond

2013-11-22 Thread Andrea Mazzoleni
Hi Piergiorgio, How about par2? How does this work? I checked the matrix they use, and sometimes it contains some singular square submatrix. It seems that in GF(2^16) these cases are just less common. Maybe they were just unnoticed. Anyway, this seems to be an already known problem for PAR2,

Re: Triple parity and beyond

2013-11-24 Thread Andrea Mazzoleni
...@nexgo.de wrote: Hi Andrea, On Sat, Nov 23, 2013 at 08:55:08AM +0100, Andrea Mazzoleni wrote: Hi Piergiorgio, How about par2? How does this work? I checked the matrix they use, and sometimes it contains some singular square submatrix. It seems that in GF(2^16) these cases are just less common

[RFC v2 0/2] New RAID library supporting up to six parities

2014-01-06 Thread Andrea Mazzoleni
to inlined functions. - Fixes some more checkpatch.pl warnings. - Other minor style/comment changes. Andrea Mazzoleni (2): lib: raid: New RAID library supporting up to six parities fs: btrfs: Extends btrfs/raid56 to support up to six parities fs/btrfs/Kconfig |1 + fs/btrfs

[RFC v2 2/2] fs: btrfs: Extends btrfs/raid56 to support up to six parities

2014-01-06 Thread Andrea Mazzoleni
() functions are used to handle with parity instead of the old xor/raid6 ones. Signed-off-by: Andrea Mazzoleni amadva...@gmail.com --- fs/btrfs/Kconfig | 1 + fs/btrfs/raid56.c | 278 ++--- fs/btrfs/raid56.h | 12 ++- fs/btrfs/volumes.c | 4

Re: [RFC] lib: raid: New RAID library supporting up to six parities

2014-01-07 Thread Andrea Mazzoleni
Hi Neil, On 01/07, NeilBrown wrote: To do the same with up to six failures, it's now required some kind of sort function. So I would probably just make sure we always process the block is the right order. Then sorting would be irrelevant. But as I say, I haven't fiddled with the code,

Re: [RFC v2 2/2] fs: btrfs: Extends btrfs/raid56 to support up to six parities

2014-01-07 Thread Andrea Mazzoleni
Hi Chris, On 01/06, Chris Mason wrote: Neat. The faila/failb were always my least favorite part of the btrfs code ;) Did you test just raid5/6 or also the higher parity counts? At this stage no real testing was made with btrfs. The intention of this btrfs patch is mainly to get feedback on

Re: [RFC v2 0/2] New RAID library supporting up to six parities

2014-01-07 Thread Andrea Mazzoleni
On 01/06, joystick wrote: Just by looking at the Subjects, it seems patch number 0/1 is missing. It might have not gotten through to the lists, or be a numbering mistake. The patch files can be also downloaded from: http://snapraid.sourceforge.net/linux/v2/ Sorry about that, Does your code

Re: [RFC v2 0/2] New RAID library supporting up to six parities

2014-01-07 Thread Andrea Mazzoleni
Hi, It seems that the patch was to big for some linux lists. If you miss some patch files, you can download them also at: http://snapraid.sourceforge.net/linux/v2/ Sorry about that. Ciao, Andrea -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to

[RFC v3 0/3] New RAID library supporting up to six parities

2014-01-10 Thread Andrea Mazzoleni
as they are not tested. They are only example code to show how the new raid library could be integrated in existing code. Please let me know what do you think. Any kind of feedback is welcome. Thanks, Andrea Andrea Mazzoleni (3): lib: raid: New RAID library supporting up to six parities fs: btrfs

[RFC v3 3/3] crypto: async_tx: Extends crypto/async_tx to support up to six parities

2014-01-10 Thread Andrea Mazzoleni
be integrated in existing code. Signed-off-by: Andrea Mazzoleni amadva...@gmail.com --- crypto/async_tx/async_pq.c | 257 +++- crypto/async_tx/async_raid6_recov.c | 286 +--- drivers/md/Kconfig | 1 + drivers/md/raid5.c

[RFC v3 2/3] fs: btrfs: Extends btrfs/raid56 to support up to six parities

2014-01-10 Thread Andrea Mazzoleni
are now used to handle all the RAID6 P/Q logic. For kernel 3.13-rc4. WARNING! This patch is not tested, and it's NOT meant for inclusion at this stage. It's only example code to show how the new raid library could be integrated in existing code. Signed-off-by: Andrea Mazzoleni amadva...@gmail.com

[RFC v4 0/3] lib: raid: New RAID library supporting up to six parities

2014-01-25 Thread Andrea Mazzoleni
checkpatch.pl warnings. - Other minor style/comment changes. Andrea Mazzoleni (3): lib: raid: New RAID library supporting up to six parities fs: btrfs: Extends btrfs/raid56 to support up to six parities crypto: async_tx: Extends crypto/async_tx to support up to six parities crypto

[RFC v4 2/3] fs: btrfs: Extends btrfs/raid56 to support up to six parities

2014-01-25 Thread Andrea Mazzoleni
are now used to handle all the RAID6 P/Q logic. For kernel 3.13. WARNING! This patch is not tested, and it's NOT meant for inclusion at this stage. It's only example code to show how the new raid library could be integrated in existing code. Signed-off-by: Andrea Mazzoleni amadva...@gmail.com

[RFC v4 3/3] crypto: async_tx: Extends crypto/async_tx to support up to six parities

2014-01-25 Thread Andrea Mazzoleni
be integrated in existing code. Signed-off-by: Andrea Mazzoleni amadva...@gmail.com --- crypto/async_tx/async_pq.c | 257 +++- crypto/async_tx/async_raid6_recov.c | 286 +--- drivers/md/Kconfig | 1 + drivers/md/raid5