On 06/11/2014 10:34 AM, Stefan G. Weichinger wrote: > Am 11.06.2014 11:19, schrieb thegeezer: > >> Hi Stefan, >> block size / stripe size mismatches only really penalise random io, if >> you are trying to use dd and have slow speeds this would suggest >> something else is awry. >> I don't know the c600 rad chip personally, but in trying to google it it >> appears to be a motherboard based raid device? is it a real raid or >> fakeraid? >> >> I'm a little confused over your setup to help. i'm sorry if there is >> duplication but it would be useful to have all info in one hit rather >> than trying to piece it together from all your messages. > OK, will do ... > > >> 1. please can you list your hardware raid config. I'm looking for the >> physical disk sizes, the virtual disks and their raid types. do you have >> cache enabled on the raid card, is there a background scrub or anything >> like that running? do you have active seek/ prefetch configured ? >> parity size being 50% of total size is just odd to me - but i guess >> these are mirrors ? but it says raid-level3 --- just odd, most setups >> use raid0(not raid) raid1(mirror) raid5(parity stripe) raid6 (double >> parity stripe) or combinations, like 50.. raid3 is allocating a single >> disk to parity but is very rarely used. > Basically 3 RAID-6 hw-raids over 6 SAS hdds.
OK so i'm confused again. RAID6 requires minimum of 4 drives. if you have 3 raid6's then you would need 12 drives (coffee hasn't quite activated in me yet so my maths may not be right) or do you have essentially the first part of each of the six drives be virtual disk 1, the second part of each of the six drives virtual disk 2 and the third part be virtual disk 3 -- if this is the case bear in mind that the slowest part of the disk is the end of the disk -- so you are essentially hobbling your virtual disk3 but only a little, instead of being around 150MB/sec it might run at 80. you might also like to try a simple test of the following (yes lvs count as block devices) # hdparm -t /dev/sda # hdparm -t /dev/sdb # hdparm -t /dev/sdc # hdparm -t /dev/vg01/winserver_disk0 # hdparm -t /dev/vg01/amhold > I don't know where this RAID-3 term comes from -> > > # megacli -LDInfo -Lall -aALL > > > Adapter 0 -- Virtual Drive Information: > Virtual Drive: 0 (Target Id: 0) > Name :root > RAID Level : Primary-6, Secondary-3, RAID Level Qualifier-3 > Size : 500.0 GB > Sector Size : 512 > Is VD emulated : No > Parity Size : 250.0 GB > State : Optimal > Strip Size : 256 KB > Number Of Drives : 6 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Disabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 1 (Target Id: 1) > Name :swap > RAID Level : Primary-6, Secondary-3, RAID Level Qualifier-3 > Size : 8.0 GB > Sector Size : 512 > Is VD emulated : No > Parity Size : 4.0 GB > State : Optimal > Strip Size : 64 KB > Number Of Drives : 6 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAheadNone, Cached, No Write Cache > if Bad BBU > Current Cache Policy: WriteBack, ReadAheadNone, Cached, No Write Cache > if Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Disabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 2 (Target Id: 2) > Name :lvm > RAID Level : Primary-6, Secondary-3, RAID Level Qualifier-3 > Size : 1.321 TB > Sector Size : 512 > Is VD emulated : No > Parity Size : 676.5 GB > State : Optimal > Strip Size : 64 KB > Number Of Drives : 6 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Disabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > > >> 2. how many other devices are actively doing IO? do you have any other >> raid cards/io cards of note that might be clashign on the board. > The Intel C600 Controller seems to only run the LTO-4-drive in the > server while the > > LSI Logic / Symbios Logic MegaRAID SAS 2108 > > runs the 6 hard disks. > > # lspci > 00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07) > 00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express > Root Port 1a (rev 07) > 00:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express > Root Port 1b (rev 07) > 00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express > Root Port 3a in PCI Express Mode (rev 07) > 00:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address > Map, VTd_Misc, System Management (rev 07) > 00:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control > Status and Global Errors (rev 07) > 00:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07) > 00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI > Express Virtual Root Port (rev 05) > 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 > Enhanced Host Controller #2 (rev 05) > 00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI > Express Root Port 1 (rev b5) > 00:1c.4 PCI bridge: Intel Corporation C600/X79 series chipset PCI > Express Root Port 5 (rev b5) > 00:1c.5 PCI bridge: Intel Corporation C600/X79 series chipset PCI > Express Root Port 6 (rev b5) > 00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI > Express Root Port 8 (rev b5) > 00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 > Enhanced Host Controller #1 (rev 05) > 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) > 00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC > Controller (rev 05) > 00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset > 6-Port SATA AHCI Controller (rev 05) > 00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host > Controller (rev 05) > 02:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 > [Liberator] (rev 05) > 04:00.0 Serial Attached SCSI controller: Intel Corporation C604/X79 > series chipset 4-Port SATA/SAS Storage Control Unit (rev 05) > 04:00.3 SMBus: Intel Corporation C600/X79 series chipset SMBus > Controller 0 (rev 05) > 06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network > Connection > 07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network > Connection > 08:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA > G200e [Pilot] ServerEngines (SEP1) (rev 05) > ff:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 > (rev 07) > ff:08.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link > Reut 0 (rev 07) > ff:08.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link > Reut 0 (rev 07) > ff:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 > (rev 07) > ff:09.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link > Reut 1 (rev 07) > ff:09.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link > Reut 1 (rev 07) > ff:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power > Control Unit 0 (rev 07) > ff:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power > Control Unit 1 (rev 07) > ff:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power > Control Unit 2 (rev 07) > ff:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power > Control Unit 3 (rev 07) > ff:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt > Control Registers (rev 07) > ff:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore > and Scratchpad Configuration Registers (rev 07) > ff:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast > Register 0 (rev 07) > ff:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast > Register 0 (rev 07) > ff:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller System Address Decoder 0 (rev 07) > ff:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System > Address Decoder (rev 07) > ff:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast > Register 0 (rev 07) > ff:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast > Register 0 (rev 07) > ff:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller System Address Decoder 1 (rev 07) > ff:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor > Home Agent (rev 07) > ff:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 > Processor Home Agent Performance Monitoring (rev 07) > ff:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller Registers (rev 07) > ff:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller RAS Registers (rev 07) > ff:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller Target Address Decoder 0 (rev 07) > ff:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller Target Address Decoder 1 (rev 07) > ff:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller Target Address Decoder 2 (rev 07) > ff:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller Target Address Decoder 3 (rev 07) > ff:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller Target Address Decoder 4 (rev 07) > ff:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller Channel 0-3 Thermal Control 0 (rev 07) > ff:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller Channel 0-3 Thermal Control 1 (rev 07) > ff:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller ERROR Registers 0 (rev 07) > ff:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller ERROR Registers 1 (rev 07) > ff:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller Channel 0-3 Thermal Control 3 (rev 07) > ff:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller ERROR Registers 2 (rev 07) > ff:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated > Memory Controller ERROR Registers 3 (rev 07) > ff:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07) > ff:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07) > ff:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to > PCI Express Performance Monitor (rev 07) > ff:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 > QuickPath Interconnect Agent Ring Registers (rev 07) > ff:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to > QuickPath Interconnect Link 0 Performance Monitor (rev 07) > ff:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to > QuickPath Interconnect Link 1 Performance Monitor (rev 07) > i notice the core i7 only now. have you disabled turbo boost in the bios ? this is great for a desktop but awful for a server as it disables all those extra cores for a single busy thread >> 3. do you have active I/O when doing your performance tests? if you have >> several virtual machines running depending on what they are doing they >> will crucify your access. > My testing was done with all the VMs stopped! eek > > >> 4. are you using any type of CGroups ? > Yes. systemd needs them. > I attach the kernel-config. ok it would be useful to know how the storage cgroups are configured, as this will also hobble things. cgroups are a great way of limiting or guaranteeing performance. by default i believe systemd will aim for user interactivity, but you want to change that to be more balanced. maybe some else can suggest how best to configure systemd cgroups. meanwhile can you # tree /sys/fs/cgroup/ >> 5. i'm also confused over your LVM config. please can you send the >> output of "vgs" "pvs" and "lvs -a -o +devices" > /dev/sdc is the third hardware raid array listed above and used as PV > for the VG vg01 -> > > # pvs > PV VG Fmt Attr PSize PFree > /dev/sdc vg01 lvm2 a-- 1,32t 708,86g > > # vgs > VG #PV #LV #SN Attr VSize VFree > vg01 1 2 0 wz--n- 1,32t 708,86g > > # lvs -a -o +devices > LV VG Attr LSize Pool Origin Data% Move Log > Cpy%Sync Convert Devices > amhold vg01 -wi-ao---- 400,00g > /dev/sdc(0) > winserver_disk0 vg01 -wi-ao---- 244,14g > /dev/sdc(102400) > > >> 6. please also send the output of "mount" > # mount > proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) > sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) > devtmpfs on /dev type devtmpfs > (rw,nosuid,size=8203540k,nr_inodes=2050885,mode=755) > tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) > devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620) > tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) > tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) > cgroup on /sys/fs/cgroup/systemd type cgroup > (rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) > cgroup on /sys/fs/cgroup/cpuset type cgroup > (rw,nosuid,nodev,noexec,relatime,cpuset) > cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup > (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu) > cgroup on /sys/fs/cgroup/devices type cgroup > (rw,nosuid,nodev,noexec,relatime,devices) > /dev/sda on / type btrfs (rw,noatime,space_cache) > mqueue on /dev/mqueue type mqueue (rw,relatime) > tmpfs on /tmp type tmpfs (rw) > tmpfs on /var/tmp/portage type tmpfs (rw,relatime) > /dev/mapper/vg01-amhold on /mnt/amhold type xfs > (rw,noatime,attr2,inode64,noquota) > tmpfs on /run/user/0 type tmpfs > (rw,nosuid,nodev,relatime,size=1641324k,mode=700) > /dev/sda on /mnt/btrfs_windows type btrfs (rw,noatime,space_cache) a little odd that you have no partitions but otherwise minor. > > the last line is meant as a target directory for dumping the content of > the LV winserver_disk0 ... it is a btrfs subvolume mounted with > compression turned OFF. > if you consider that your drives are on the same raidset, they are all essentially "one disk". copying from one disk to the same disk you are halving the speed it will work at. > >> 7. do you have atop or iotop that you can use to monitor performance - >> specifically we are looking for disk ios per device and disk latency per >> device. both before and during you are trying to run your backup. > I will install atop and check ... > >> this should give us a better idea of where the problems lay. > Thanks so far! > > Stefan > just some extra thoughts
