Hi all!
I get some trouble using mdadm over LVM2. Here is my config:
I've 2 servers (xen1 and xen2 - their hostnames in my local network)
with configuration below:
Each server have 4 SATA disks, 1 Tb each attached to motherboard.
4x4 Gb ddr3
debian squeeze x64 installed:
root@xen2:~# uname -a
Linux xen2 2.6.32-5-xen-amd64 #1 SMP Wed Jan 12 05:46:49 UTC 2011 x86_64
GNU/Linux
Storage configuration:
First 256 Mb and second 32 Gb of 2 of 4 disks are used for raid1 devices
for /boot and swap respectively.
The rest of space, 970 Gb on all 4 sata disks are used as raid10.
There is LVM2 installed over that raid10. Volume group is named xenlvm
(that servers are expected to use as xen 4.0.1 hosts, but the story is
not about Xen troubles).
/ , /var, /home are located on logical volumes of small size:
root@xen2:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/XENLVM-home
9.2G 6.0G 2.8G 69% /
tmpfs 7.6G 0 7.6G 0% /lib/init/rw
udev 7.1G 316K 7.1G 1% /dev
tmpfs 7.6G 0 7.6G 0% /dev/shm
/dev/md3 223M 31M 180M 15% /boot
/dev/mapper/XENLVM-var
9.2G 150M 8.6G 2% /home
/dev/mapper/XENLVM-root
9.2G 2.5G 6.3G 29% /var
About 900 Gb on "xenlvm" volume group are left free to create new
logical volumes, which are expected to use as block devices for raid1
partitions. One member of such array is local logical volume and the
second is an Ata over Ethernet device.
It's name (of this aoe device) is e.g. e0.1.
We need such complications to run Xen vm-s. Our vm-s use raid1 devices
for storing their data. And if one of two hosts (xen1 or xen2) die with
catastrofic failure the second xen host will hold virtual machine block
device so we can start it.
This two servers have 2 ethernet devices on each. One of eth dev (eth1
on each) is comunicating with our lan (to connect to the server). The
second ethernet device (eth0 on each) is connected with another server
using ethernet cross connection with 1 Gbit/s throughput to provide disk
space via ata over ethernet.
So here is the problem with this RAID1 device:
I've configured one 20Gigs RAID1 so it have 20GiGs AoE device and 20
GiGs LVM local block storage.
mdadm -C /dev/md3 --level=1 --raid-devices=2 /dev/etherd/e0.1
/dev/xenlvm/raid20gig
And installed Windows 2003 over this volume. Made some configurations
inside and installed some soft. Then I backed up image of this volume
using dd:
dd if=/dev/md3 of=/backups/md3_date.dd
Then I decided to run "more of this" virtual machines from that backup.
So I created another one raid1 device with 20 Gigs capacity:
mdadm -C /dev/md4 --level=1 --raid-devices=2 /dev/xenlvm/raid20gig2
/dev/etherd/e0.2
And wrote that dd backup to it:
dd if=/backups/md3_date.dd of=/dev/md4.
And started domU with this md4 device as hdd. It runs smoothly. But
when I look at
cat /proc/mdstat I see that one of backed deviced is in faulting state:
md4 : active raid1 dm-15[0](F) etherd/e1.5[1]
20970424 blocks super 1.2 [2/1] [_U]
that dm-15 is the LVM2 device /dev/xenlvm/raid20gig2
If I hot remove, re-add failing device the raid volume begins to resync
as it was in normal state :
root@xen1:~# mdadm /dev/md4 -r /dev/dm-15
mdadm: hot removed /dev/dm-15 from /dev/md8
root@xen1:~# mdadm /dev/md4 -a /dev/dm-15
mdadm: re-added /dev/dm-15
Only faulting device listing showed below, as I said, there is raid10
in system, and there's no problem with it.
root@xen1:~# cat /proc/mdstat
Personalities : [raid1] [raid10]
md4 : active raid1 dm-15[0] etherd/e1.5[1]
20970424 blocks super 1.2 [2/1] [_U]
[>....................] recovery = 1.0% (218752/20970424) finish=17.3min
speed=19886K/sec
So I started to watch /var/log/syslog and messages for some errors and
found a message bellow:
*raid10_make_request bug: can't convert block across chunks or bigger
than 512k 965198847 4
*
This message appears in the log at the moment when state of lvm block
device dm-15 changes from normal to faulting in /proc/mdstat.
That's not the end of the story. I saw this message on xen1 host, so
that was local lvm device. But at some moment this problem appeared on
the second host - xen2.
And this message appears in xen2 /var/log/kern.log and floods it very
fast, so I get my /var full in two days. And after that my aoe-device on
xen1 gets into "down" state and the vm dies.
I googled for this error and found only posts about some redhat and
debian etch kernel bug in year 2007-2009.