Satoru Takeuchi <takeuchi_sat...@jp.fujitsu.com> wrote: > > Let me ask some questions.
Sure - thanks for taking an interest. > On 2014/12/17 11:42, Charles Cazabon wrote: > > There's roughly 16TB of data in this filesystem (the filesystem is ~17TB). > > The btrfs filesystem is a simple single volume, no snapshots, multiple > > devices, or anything like that. It's an LVM logical volume on top of > > dmcrypt on top of an mdadm RAID set (8 disks in RAID 6). > > Q1. You mean your Btrfs file system exists on the top of > the following deep layers? > > +---------------+ > |Btrfs(single) | > +---------------+ > |LVM(non RAID?) | > +---------------+ > |dmcrypt | > +---------------+ > |mdadm RAID set | > +---------------+ Yes, precisely. mdadm is used to make a large RAID6 device, which is encrypted with LUKS, on top of which is layered LVM (for ease of management), and the btrfs filesystem sits on that. > Q2. If Q1 is true, is it possible to reduce that layers as follows? > > +-----------+ > |Btrfs(*1) | > +-----------+ > |dmcrypt | > +-----------+ I don't see how I could do that - I simply have far too much data for a single disk (not to mention I don't want to risk loss of data from a single disk failing). This filesystem has 16.x TB of data in it at present. > It's because there are too many layers and these have > the same/similar features and heavy layered file system > tends to cause more trouble than thinner layered ones > regardless of file system type. This configuration is one I've been using for many years. It's only recently that I've noticed it being particularly slow with btrfs -- I don't know if that's because the filesystem has filled up past some critical point, or due to something else entirely. That's why I'm trying to figure this out. > *1) Currently I don't recommend you to use RAID56 of Btrfs. > So, if RAID6 is mandatory, mdadm RAID6 is also necessary. Yes, exactly. That's why I use mdadm. > > The speeds I'm seeing (with iotop) fluctuate a lot. They spend most of > > the time in the range of 1-3 MB/s, with large periods of time where no IO > > seems to happen at all, and occasional short spikes to ~25-30 MB/s. > > System load seems to sit around 10-12 (with only 2 processes reported as > > running, everything else sleeping) while this happens. [...] > > Other filesystems on the same physical disks have no trouble exceeding > > 100MB/s reads. The machine is not swapping (16GB RAM, ~8GB swap with 0 > > swap used). > > Q3. They are also consist of the following layers? Yes, exactly the same configuration. The fact that I don't see any speed problems with other filesystems (even in the same LVM volume group) leads me in the direction of suspecting something to do with btrfs. > Q4. Are other filesystems also near-full? No, not particularly. Now, the btrfs volume in question isn't exactly close to full - there's more than 500 GB free. It's just *relatively* full. > Q5. Is there any error/warning message about > Btrfs/LVM/dmcrypt/mdadm/hardwares? No, no errors or warnings in logs related to the disks, LVM, or btrfs. I have historically, with previous kernels, gotten the "task blocked for more than 120 seconds" warnings fairly often, but I haven't seen those lately. Is there any other info I can collect on this that would help? Thanks, Charles -- ----------------------------------------------------------------------- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ ----------------------------------------------------------------------- -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
