Hi Marcus, Good day to you, and thank you for your e-mail. See my reply inline.
On Fri, Oct 4, 2013 at 12:29 AM, Marcus Sorensen <shadow...@gmail.com>wrote: > Are you using the release artifacts, or your own 4.2 build? > [Indra:] We are using the release artifacts from below repo since we are using Ubuntu: deb http://cloudstack.apt-get.eu/ubuntu precise 4.2 > When you rebooted the host, did the problem go away or come back the > same? [Indra:] When I rebooted the host, the problem go away for a while, but it will come back again after some time. It will come randomly at the time when we need to create a new instance on that host, or start an existing stopped instance. > You may want to look at 'virsh pool-list' to see if libvirt is > mounting/registering the secondary storage. > [Indra:] This is the result of the virsh pool-list command: root@hv-kvm-02:/var/log/libvirt# virsh pool-list Name State Autostart ----------------------------------------- 301071ac-4c1d-4eac-855b-124126da0a38 active no 5230667e-9c58-3ff6-983c-5fc2a72df669 active no d433809b-01ea-3947-ba0f-48077244e4d6 active no Strange thing is that none of my secondary storage IDs are there. Could it be that the ID might have changed during Cloudstack upgrade? Here is the list of my secondary storage (there are two of them) even though they are on the same NFS server: c02da448-b9f4-401b-b8d5-83e8ead5cfdenfs://10.237.11.31/mnt/vol1/sec-storage NFS 5937edb6-2e95-4ae2-907b-80fe4599ed87nfs:// 10.237.11.31/mnt/vol1/sec-storage2NFS > Is this happening on multiple hosts, the same way? [Indra:] Yes, it is happening on all the two hosts that I have, the same way. > You may want to > look at /etc/mtab, if the system reports it's mounted, though it's > not, it might be in there. Look at /proc/mounts as well. > [Indra:] Please find result of df, /etc/mtab and /proc/mounts below. The "ghost" mount point is on df and /etc/mtab, but not on /proc/mounts. root@hv-kvm-02:/etc# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 4195924 4192372 0 100% / udev 132053356 4 132053352 1% /dev tmpfs 52825052 704 52824348 1% /run none 5120 0 5120 0% /run/lock none 132062620 0 132062620 0% /run/shm cgroup 132062620 0 132062620 0% /sys/fs/cgroup /dev/sda6 10650544 2500460 7609056 25% /home 10.237.11.31:/mnt/vol1/sec-storage2/template/tmpl/2/288 4195924 4192372 0 100% /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 root@hv-kvm-02:/etc# cat /etc/mtab /dev/sda1 / ext4 rw,errors=remount-ro 0 0 proc /proc proc rw,noexec,nosuid,nodev 0 0 sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0 none /sys/fs/fuse/connections fusectl rw 0 0 none /sys/kernel/debug debugfs rw 0 0 none /sys/kernel/security securityfs rw 0 0 udev /dev devtmpfs rw,mode=0755 0 0 devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=0620 0 0 tmpfs /run tmpfs rw,noexec,nosuid,size=10%,mode=0755 0 0 none /run/lock tmpfs rw,noexec,nosuid,nodev,size=5242880 0 0 none /run/shm tmpfs rw,nosuid,nodev 0 0 cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0 cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0 /dev/sda6 /home ext4 rw 0 0 rpc_pipefs /run/rpc_pipefs rpc_pipefs rw 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,noexec,nosuid,nodev 0 0 10.237.11.31:/mnt/vol1/sec-storage2/template/tmpl/2/288 /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 nfs rw,addr=10.237.11.31 0 0 rootfs / rootfs rw 0 0 sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,relatime,size=132053356k,nr_inodes=33013339,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,relatime,size=52825052k,mode=755 0 0 /dev/disk/by-uuid/d21f4676-075d-44be-a378-92ef6aaf2496 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0 none /sys/fs/fuse/connections fusectl rw,relatime 0 0 none /sys/kernel/debug debugfs rw,relatime 0 0 cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0 none /sys/kernel/security securityfs rw,relatime 0 0 none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0 cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0 /dev/sda6 /home ext4 rw,relatime,data=ordered 0 0 rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 If I tried to un-mount (umount) the "ghost" mount point, it will fail with below error message saying that it is not found in /proc/mounts: root@hv-kvm-02:/etc# umount /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 was not found in /proc/mounts /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 was not found in /proc/mounts Do you have any idea what could be the problem? Is it libvirt issue or CloudStack issue? Any help on this is greatly appreciated. Since I am upgrading a production setup, we only have 1-2 hours left to resolve the problem, otherwise we will be forced to revert back to 4.1.1 (for the fourth time). :( Urgently looking forward to your reply, thank you. Cheers. > > On Thu, Oct 3, 2013 at 9:53 AM, Indra Pramana <in...@sg.or.id> wrote: > > Dear all, > > > > We face a major problem after upgrading to 4.2.0. Mounting from KVM hosts > > to secondary storage seems to fail, every time a new VM instance is > > created, it will use up the / (root) partition of the KVM hosts instead. > > > > Here is the df result: > > > > ==== > > root@hv-kvm-02:/home/indra# df > > Filesystem 1K-blocks Used > > Available Use% Mounted on > > /dev/sda1 4195924 > > 4195924 0 100% / > > udev 132053356 4 > > 132053352 1% /dev > > tmpfs 52825052 440 > > 52824612 1% /run > > none 5120 > > 0 5120 0% /run/lock > > none 132062620 0 > > 132062620 0% /run/shm > > cgroup 132062620 0 > > 132062620 0% /sys/fs/cgroup > > /dev/sda6 10650544 2500424 > > 7609092 25% /home > > 10.237.11.31:/mnt/vol1/sec-storage2/template/tmpl/2/288 4195924 > > 4195924 0 100% /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 > > ==== > > > > The strange thing is that df shows that it seems to be mounted, but it's > > actually not mounted. If you noticed, the total capacity of the mount > point > > is exactly the same as the capacity of the / (root) partition. By right > it > > should show 7 TB instead of just 4 GB. > > > > This caused VM creation to be error due to out of disk space. This also > > affect the KVM operations since the / (root) partition becomes full, and > we > > can only release the space after we reboot the KVM host. > > > > Anyone experience this problem before? We are at loss on how to resolve > the > > problem. > > > > Looking forward to your reply, thank you. > > > > Cheers. >
