1 week to rebuid 4x 3TB raid10 is a long time!

2014-07-20 Thread TM
Hi, I have a raid10 with 4x 3TB disks on a microserver http://n40l.wikia.com/wiki/Base_Hardware_N54L , 8Gb RAM Recently one disk started to fail (smart errors), so I replaced it Mounted as degraded, added new disk, removed old Started yesterday I am monitoring /var/log/messages and it seems it wi

Re: ENOSPC errors during balance

2014-07-20 Thread Marc Joliet
Am Sat, 19 Jul 2014 19:11:00 -0600 schrieb Chris Murphy : > I'm seeing this also in the 2nd dmesg: > > [ 249.893310] BTRFS error (device sdg2): free space inode generation (0) did > not match free space cache generation (26286) > > > So you could try umounting the volume. And doing a one time

Re: ENOSPC errors during balance

2014-07-20 Thread Marc Joliet
Am Sat, 19 Jul 2014 18:53:03 -0600 schrieb Chris Murphy : > > On Jul 19, 2014, at 2:58 PM, Marc Joliet wrote: > > > Am Sat, 19 Jul 2014 22:10:51 +0200 > > schrieb Marc Joliet : > > > > [...] > >> Another random idea: the number of errors decreased the second time I ran > >> balance (from 4 to

Re: ENOSPC errors during balance

2014-07-20 Thread Marc Joliet
Am Sun, 20 Jul 2014 02:39:27 + (UTC) schrieb Duncan <1i5t5.dun...@cox.net>: > Chris Murphy posted on Sat, 19 Jul 2014 11:38:08 -0600 as excerpted: > > > I'm not sure of the reason for the "BTRFS info (device sdg2): 2 enospc > > errors during balance" but it seems informational rather than eit

Re: ENOSPC errors during balance

2014-07-20 Thread Marc Joliet
Am Sun, 20 Jul 2014 12:22:33 +0200 schrieb Marc Joliet : [...] > I'll try this and see, but I think I have more files >1GB than would account > for this error (which comes towards the end of the balance when only a few > chunks are left). I'll see what "find /mnt -type f -size +1G" finds :) . No

Re: ENOSPC errors during balance

2014-07-20 Thread Duncan
Marc Joliet posted on Sun, 20 Jul 2014 12:22:33 +0200 as excerpted: > On the other hand, the wiki [0] says that defragmentation (and > balancing) is optional, and the only reason stated for doing either is > because they "will have impact on performance". Yes. That's what threw off the other guy

Re: 1 week to rebuid 4x 3TB raid10 is a long time!

2014-07-20 Thread Duncan
TM posted on Sun, 20 Jul 2014 08:45:51 + as excerpted: > One week for a raid10 rebuild 4x3TB drives is a very long time. > Any thoughts? > Can you share any statistics from your RAID10 rebuilds? Well, 3 TB is big and spinning rust is slow. Even using the smaller power-of-10 (1000) figures f

Re: 1 week to rebuid 4x 3TB raid10 is a long time!

2014-07-20 Thread Tomasz Torcz
On Sun, Jul 20, 2014 at 01:53:34PM +, Duncan wrote: > TM posted on Sun, 20 Jul 2014 08:45:51 + as excerpted: > > > One week for a raid10 rebuild 4x3TB drives is a very long time. > > Any thoughts? > > Can you share any statistics from your RAID10 rebuilds? > > > At a week, that's nearly

[GIT PULL] Btrfs

2014-07-20 Thread Chris Mason
Hi Linus, We have two more fixes in my for-linus branch: git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git for-linus I was hoping to also include a fix for a btrfs deadlock with compression enabled, but we're still nailing that one down. Liu Bo (1) commits (+11/-0): Btrfs

Re: 1 week to rebuid 4x 3TB raid10 is a long time!

2014-07-20 Thread Austin S Hemmelgarn
On 07/20/2014 10:00 AM, Tomasz Torcz wrote: > On Sun, Jul 20, 2014 at 01:53:34PM +, Duncan wrote: >> TM posted on Sun, 20 Jul 2014 08:45:51 + as excerpted: >> >>> One week for a raid10 rebuild 4x3TB drives is a very long time. >>> Any thoughts? >>> Can you share any statistics from your RAI

Re: Questions on incremental backups

2014-07-20 Thread Sam Bull
Thanks everyone for the responses. I'll start setting up my backup strategy in 2 or 3 weeks. I'll give the diff and unionFS tips a go, and report back on any progress. signature.asc Description: This is a digitally signed message part

Re: 1 week to rebuid 4x 3TB raid10 is a long time!

2014-07-20 Thread ashford
Tomasz, > On Sun, Jul 20, 2014 at 01:53:34PM +, Duncan wrote: >> TM posted on Sun, 20 Jul 2014 08:45:51 + as excerpted: >> >>> One week for a raid10 rebuild 4x3TB drives is a very long time. >>> Any thoughts? >>> Can you share any statistics from your RAID10 rebuilds? >> >> At a week, that

Re: 1 week to rebuid 4x 3TB raid10 is a long time!

2014-07-20 Thread TM
whisperpc.com> writes: > > Finally, TM didn't mention anything about other I/O activity on the array, > which, regardless of the method of reconstruction, could have a > significant impact on the speed of a reconstruction. > > There are a LOT of parameters here that could impact throughput. S

Re: 1 week to rebuid 4x 3TB raid10 is a long time!

2014-07-20 Thread TM
whisperpc.com> writes: > > Finally, TM didn't mention anything about other I/O activity on the array, > which, regardless of the method of reconstruction, could have a > significant impact on the speed of a reconstruction. > > There are a LOT of parameters here that could impact throughput. S

Re: 1 week to rebuid 4x 3TB raid10 is a long time!

2014-07-20 Thread Bob Marley
On 20/07/2014 10:45, TM wrote: Hi, I have a raid10 with 4x 3TB disks on a microserver http://n40l.wikia.com/wiki/Base_Hardware_N54L , 8Gb RAM Recently one disk started to fail (smart errors), so I replaced it Mounted as degraded, added new disk, removed old Started yesterday I am monitoring /va

Re: 1 week to rebuid 4x 3TB raid10 is a long time!

2014-07-20 Thread Roman Mamedov
On Sun, 20 Jul 2014 21:15:31 +0200 Bob Marley wrote: > Hi TM, are you doing other significant filesystem activity during this > rebuild, especially random accesses? > This can reduce performances a lot on HDDs. > E.g. if you were doing strenous multithreaded random writes in the > meanwhile, I

Re: ENOSPC errors during balance

2014-07-20 Thread Marc Joliet
Am Sun, 20 Jul 2014 13:40:54 +0200 schrieb Marc Joliet : > Am Sun, 20 Jul 2014 12:22:33 +0200 > schrieb Marc Joliet : > > [...] > > I'll try this and see, but I think I have more files >1GB than would account > > for this error (which comes towards the end of the balance when only a few > > chunk

Re: ENOSPC errors during balance

2014-07-20 Thread Marc Joliet
Oh, and because I'm forgetful, here the new dmesg output. The new content (relative to dmesg4) starts at line 2513. -- Marc Joliet -- "People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup dmesg5.log.xz Description: application/xz signature.asc

Re: 1 week to rebuid 4x 3TB raid10 is a long time!

2014-07-20 Thread ashford
This is the cause for the slow reconstruct. > I believe the problem here might be that a Btrfs rebuild *is* a strenuous > random read (+ random-ish write) just by itself. If you assume a 12ms average seek time (normal for 7200RPM SATA drives), an 8.3ms rotational latency (half a rotation), an ave

Re: 1 week to rebuid 4x 3TB raid10 is a long time!

2014-07-20 Thread Bob Marley
On 20/07/2014 21:36, Roman Mamedov wrote: On Sun, 20 Jul 2014 21:15:31 +0200 Bob Marley wrote: Hi TM, are you doing other significant filesystem activity during this rebuild, especially random accesses? This can reduce performances a lot on HDDs. E.g. if you were doing strenous multithreaded r

Re: Blocked tasks on 3.15.1

2014-07-20 Thread Matt
>[ deadlocks during rsync in 3.15 with compression enabled ] >Hi everyone, >I still haven't been able to reproduce this one here, but I'm going >through a series of tests with lzo compression foraced and every >operation forced to ordered. Hopefully it'll kick it out soon. > >While I'm hammering

Re: 1 week to rebuid 4x 3TB raid10 is a long time!

2014-07-20 Thread George Mitchell
On 07/20/2014 02:28 PM, Bob Marley wrote: On 20/07/2014 21:36, Roman Mamedov wrote: On Sun, 20 Jul 2014 21:15:31 +0200 Bob Marley wrote: Hi TM, are you doing other significant filesystem activity during this rebuild, especially random accesses? This can reduce performances a lot on HDDs. E.g.

Re: 1 week to rebuid 4x 3TB raid10 is a long time!

2014-07-20 Thread Wang Shilong
Hi, On 07/20/2014 04:45 PM, TM wrote: Hi, I have a raid10 with 4x 3TB disks on a microserver http://n40l.wikia.com/wiki/Base_Hardware_N54L , 8Gb RAM Recently one disk started to fail (smart errors), so I replaced it Mounted as degraded, added new disk, removed old Started yesterday I am monito

Re: ENOSPC errors during balance

2014-07-20 Thread Duncan
Marc Joliet posted on Sun, 20 Jul 2014 21:44:40 +0200 as excerpted: > Am Sun, 20 Jul 2014 13:40:54 +0200 schrieb Marc Joliet : > >> Am Sun, 20 Jul 2014 12:22:33 +0200 schrieb Marc Joliet : >> >> [...] >> > I'll try this and see, but I think I have more files >1GB than would >> > account for this

Re: 1 week to rebuid 4x 3TB raid10 is a long time!

2014-07-20 Thread Duncan
ashford posted on Sun, 20 Jul 2014 12:59:21 -0700 as excerpted: > If you assume a 12ms average seek time (normal for 7200RPM SATA drives), > an 8.3ms rotational latency (half a rotation), an average 64kb write and > a 100MB/S streaming write speed, each write comes in at ~21ms, which > gives us ~4