Hi Marcus and all, I also find some strange and interesting error messages from the libvirt logs:
http://pastebin.com/5ByfNpAf Looking forward to your reply, thank you. Cheers. On Fri, Oct 4, 2013 at 1:38 AM, Indra Pramana <in...@sg.or.id> wrote: > Hi Marcus, > > Good day to you, and thank you for your e-mail. See my reply inline. > > On Fri, Oct 4, 2013 at 12:29 AM, Marcus Sorensen <shadow...@gmail.com>wrote: > >> Are you using the release artifacts, or your own 4.2 build? >> > > [Indra:] We are using the release artifacts from below repo since we are > using Ubuntu: > > deb http://cloudstack.apt-get.eu/ubuntu precise 4.2 > > >> When you rebooted the host, did the problem go away or come back the >> same? > > > [Indra:] When I rebooted the host, the problem go away for a while, but it > will come back again after some time. It will come randomly at the time > when we need to create a new instance on that host, or start an existing > stopped instance. > > >> You may want to look at 'virsh pool-list' to see if libvirt is >> mounting/registering the secondary storage. >> > > [Indra:] This is the result of the virsh pool-list command: > > root@hv-kvm-02:/var/log/libvirt# virsh pool-list > Name State Autostart > ----------------------------------------- > 301071ac-4c1d-4eac-855b-124126da0a38 active no > 5230667e-9c58-3ff6-983c-5fc2a72df669 active no > d433809b-01ea-3947-ba0f-48077244e4d6 active no > > Strange thing is that none of my secondary storage IDs are there. Could it > be that the ID might have changed during Cloudstack upgrade? Here is the > list of my secondary storage (there are two of them) even though they are > on the same NFS server: > c02da448-b9f4-401b-b8d5-83e8ead5cfde nfs:// > 10.237.11.31/mnt/vol1/sec-storage NFS > 5937edb6-2e95-4ae2-907b-80fe4599ed87 nfs:// > 10.237.11.31/mnt/vol1/sec-storage2 NFS > >> Is this happening on multiple hosts, the same way? > > > [Indra:] Yes, it is happening on all the two hosts that I have, the same > way. > > >> You may want to >> look at /etc/mtab, if the system reports it's mounted, though it's >> not, it might be in there. Look at /proc/mounts as well. >> > > [Indra:] Please find result of df, /etc/mtab and /proc/mounts below. The > "ghost" mount point is on df and /etc/mtab, but not on /proc/mounts. > > root@hv-kvm-02:/etc# df > > Filesystem 1K-blocks Used > Available Use% Mounted on > /dev/sda1 4195924 > 4192372 0 100% / > > udev 132053356 4 > 132053352 1% /dev > tmpfs 52825052 704 > 52824348 1% /run > > none 5120 > 0 5120 0% /run/lock > none 132062620 0 > 132062620 0% /run/shm > cgroup 132062620 0 > 132062620 0% /sys/fs/cgroup > /dev/sda6 10650544 > 2500460 7609056 25% /home > 10.237.11.31:/mnt/vol1/sec-storage2/template/tmpl/2/288 4195924 > 4192372 0 100% /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 > > root@hv-kvm-02:/etc# cat /etc/mtab > /dev/sda1 / ext4 rw,errors=remount-ro 0 0 > proc /proc proc rw,noexec,nosuid,nodev 0 0 > sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0 > none /sys/fs/fuse/connections fusectl rw 0 0 > none /sys/kernel/debug debugfs rw 0 0 > none /sys/kernel/security securityfs rw 0 0 > udev /dev devtmpfs rw,mode=0755 0 0 > devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=0620 0 0 > tmpfs /run tmpfs rw,noexec,nosuid,size=10%,mode=0755 0 0 > none /run/lock tmpfs rw,noexec,nosuid,nodev,size=5242880 0 0 > none /run/shm tmpfs rw,nosuid,nodev 0 0 > cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0 > cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0 > cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0 > cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0 > cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0 > cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0 > cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0 > cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0 > cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0 > /dev/sda6 /home ext4 rw 0 0 > rpc_pipefs /run/rpc_pipefs rpc_pipefs rw 0 0 > binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,noexec,nosuid,nodev 0 0 > 10.237.11.31:/mnt/vol1/sec-storage2/template/tmpl/2/288 > /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 nfs rw,addr=10.237.11.31 0 0 > > rootfs / rootfs rw 0 0 > sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 > proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 > udev /dev devtmpfs rw,relatime,size=132053356k,nr_inodes=33013339,mode=755 > 0 0 > devpts /dev/pts devpts > rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 > tmpfs /run tmpfs rw,nosuid,relatime,size=52825052k,mode=755 0 0 > /dev/disk/by-uuid/d21f4676-075d-44be-a378-92ef6aaf2496 / ext4 > rw,relatime,errors=remount-ro,data=ordered 0 0 > none /sys/fs/fuse/connections fusectl rw,relatime 0 0 > none /sys/kernel/debug debugfs rw,relatime 0 0 > cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0 > none /sys/kernel/security securityfs rw,relatime 0 0 > none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 > none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0 > cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0 > cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0 > cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0 > cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0 > cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0 > cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0 > cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0 > cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0 > /dev/sda6 /home ext4 rw,relatime,data=ordered 0 0 > rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 > binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc > rw,nosuid,nodev,noexec,relatime 0 0 > > If I tried to un-mount (umount) the "ghost" mount point, it will fail with > below error message saying that it is not found in /proc/mounts: > > root@hv-kvm-02:/etc# umount /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 > /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 was not found in /proc/mounts > /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 was not found in /proc/mounts > > Do you have any idea what could be the problem? Is it libvirt issue or > CloudStack issue? > > Any help on this is greatly appreciated. Since I am upgrading a production > setup, we only have 1-2 hours left to resolve the problem, otherwise we > will be forced to revert back to 4.1.1 (for the fourth time). :( > > Urgently looking forward to your reply, thank you. > > Cheers. > > > > >> >> On Thu, Oct 3, 2013 at 9:53 AM, Indra Pramana <in...@sg.or.id> wrote: >> > Dear all, >> > >> > We face a major problem after upgrading to 4.2.0. Mounting from KVM >> hosts >> > to secondary storage seems to fail, every time a new VM instance is >> > created, it will use up the / (root) partition of the KVM hosts instead. >> > >> > Here is the df result: >> > >> > ==== >> > root@hv-kvm-02:/home/indra# df >> > Filesystem 1K-blocks >> Used >> > Available Use% Mounted on >> > /dev/sda1 4195924 >> > 4195924 0 100% / >> > udev 132053356 >> 4 >> > 132053352 1% /dev >> > tmpfs 52825052 >> 440 >> > 52824612 1% /run >> > none 5120 >> > 0 5120 0% /run/lock >> > none 132062620 >> 0 >> > 132062620 0% /run/shm >> > cgroup 132062620 >> 0 >> > 132062620 0% /sys/fs/cgroup >> > /dev/sda6 10650544 >> 2500424 >> > 7609092 25% /home >> > 10.237.11.31:/mnt/vol1/sec-storage2/template/tmpl/2/288 4195924 >> > 4195924 0 100% /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 >> > ==== >> > >> > The strange thing is that df shows that it seems to be mounted, but it's >> > actually not mounted. If you noticed, the total capacity of the mount >> point >> > is exactly the same as the capacity of the / (root) partition. By right >> it >> > should show 7 TB instead of just 4 GB. >> > >> > This caused VM creation to be error due to out of disk space. This also >> > affect the KVM operations since the / (root) partition becomes full, >> and we >> > can only release the space after we reboot the KVM host. >> > >> > Anyone experience this problem before? We are at loss on how to resolve >> the >> > problem. >> > >> > Looking forward to your reply, thank you. >> > >> > Cheers. >> > >
