It sort of looks like the out of space triggered the issue. Libvirt shows 2013-10-03 15:38:57.414+0000: 2710: error : virCommandWait:2348 : internal error Child process (/bin/umount /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669) unexpected exit status 32: error writing /etc/mtab.tmp: No space left on device
So that entry of the filesystem being mounted is orphaned in /etc/mtab, since it can't be removed. That seems to be the source of why 'df' shows the thing mounted when it isn't, at least. On Thu, Oct 3, 2013 at 11:45 AM, Indra Pramana <in...@sg.or.id> wrote: > Hi Marcus and all, > > I also find some strange and interesting error messages from the libvirt > logs: > > http://pastebin.com/5ByfNpAf > > Looking forward to your reply, thank you. > > Cheers. > > > > On Fri, Oct 4, 2013 at 1:38 AM, Indra Pramana <in...@sg.or.id> wrote: > >> Hi Marcus, >> >> Good day to you, and thank you for your e-mail. See my reply inline. >> >> On Fri, Oct 4, 2013 at 12:29 AM, Marcus Sorensen <shadow...@gmail.com>wrote: >> >>> Are you using the release artifacts, or your own 4.2 build? >>> >> >> [Indra:] We are using the release artifacts from below repo since we are >> using Ubuntu: >> >> deb http://cloudstack.apt-get.eu/ubuntu precise 4.2 >> >> >>> When you rebooted the host, did the problem go away or come back the >>> same? >> >> >> [Indra:] When I rebooted the host, the problem go away for a while, but it >> will come back again after some time. It will come randomly at the time >> when we need to create a new instance on that host, or start an existing >> stopped instance. >> >> >>> You may want to look at 'virsh pool-list' to see if libvirt is >>> mounting/registering the secondary storage. >>> >> >> [Indra:] This is the result of the virsh pool-list command: >> >> root@hv-kvm-02:/var/log/libvirt# virsh pool-list >> Name State Autostart >> ----------------------------------------- >> 301071ac-4c1d-4eac-855b-124126da0a38 active no >> 5230667e-9c58-3ff6-983c-5fc2a72df669 active no >> d433809b-01ea-3947-ba0f-48077244e4d6 active no >> >> Strange thing is that none of my secondary storage IDs are there. Could it >> be that the ID might have changed during Cloudstack upgrade? Here is the >> list of my secondary storage (there are two of them) even though they are >> on the same NFS server: >> c02da448-b9f4-401b-b8d5-83e8ead5cfde nfs:// >> 10.237.11.31/mnt/vol1/sec-storage NFS >> 5937edb6-2e95-4ae2-907b-80fe4599ed87 nfs:// >> 10.237.11.31/mnt/vol1/sec-storage2 NFS >> >>> Is this happening on multiple hosts, the same way? >> >> >> [Indra:] Yes, it is happening on all the two hosts that I have, the same >> way. >> >> >>> You may want to >>> look at /etc/mtab, if the system reports it's mounted, though it's >>> not, it might be in there. Look at /proc/mounts as well. >>> >> >> [Indra:] Please find result of df, /etc/mtab and /proc/mounts below. The >> "ghost" mount point is on df and /etc/mtab, but not on /proc/mounts. >> >> root@hv-kvm-02:/etc# df >> >> Filesystem 1K-blocks Used >> Available Use% Mounted on >> /dev/sda1 4195924 >> 4192372 0 100% / >> >> udev 132053356 4 >> 132053352 1% /dev >> tmpfs 52825052 704 >> 52824348 1% /run >> >> none 5120 >> 0 5120 0% /run/lock >> none 132062620 0 >> 132062620 0% /run/shm >> cgroup 132062620 0 >> 132062620 0% /sys/fs/cgroup >> /dev/sda6 10650544 >> 2500460 7609056 25% /home >> 10.237.11.31:/mnt/vol1/sec-storage2/template/tmpl/2/288 4195924 >> 4192372 0 100% /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 >> >> root@hv-kvm-02:/etc# cat /etc/mtab >> /dev/sda1 / ext4 rw,errors=remount-ro 0 0 >> proc /proc proc rw,noexec,nosuid,nodev 0 0 >> sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0 >> none /sys/fs/fuse/connections fusectl rw 0 0 >> none /sys/kernel/debug debugfs rw 0 0 >> none /sys/kernel/security securityfs rw 0 0 >> udev /dev devtmpfs rw,mode=0755 0 0 >> devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=0620 0 0 >> tmpfs /run tmpfs rw,noexec,nosuid,size=10%,mode=0755 0 0 >> none /run/lock tmpfs rw,noexec,nosuid,nodev,size=5242880 0 0 >> none /run/shm tmpfs rw,nosuid,nodev 0 0 >> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0 >> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0 >> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0 >> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0 >> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0 >> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0 >> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0 >> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0 >> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0 >> /dev/sda6 /home ext4 rw 0 0 >> rpc_pipefs /run/rpc_pipefs rpc_pipefs rw 0 0 >> binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,noexec,nosuid,nodev 0 0 >> 10.237.11.31:/mnt/vol1/sec-storage2/template/tmpl/2/288 >> /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 nfs rw,addr=10.237.11.31 0 0 >> >> rootfs / rootfs rw 0 0 >> sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 >> proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 >> udev /dev devtmpfs rw,relatime,size=132053356k,nr_inodes=33013339,mode=755 >> 0 0 >> devpts /dev/pts devpts >> rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 >> tmpfs /run tmpfs rw,nosuid,relatime,size=52825052k,mode=755 0 0 >> /dev/disk/by-uuid/d21f4676-075d-44be-a378-92ef6aaf2496 / ext4 >> rw,relatime,errors=remount-ro,data=ordered 0 0 >> none /sys/fs/fuse/connections fusectl rw,relatime 0 0 >> none /sys/kernel/debug debugfs rw,relatime 0 0 >> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0 >> none /sys/kernel/security securityfs rw,relatime 0 0 >> none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 >> none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0 >> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0 >> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0 >> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0 >> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0 >> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0 >> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0 >> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0 >> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0 >> /dev/sda6 /home ext4 rw,relatime,data=ordered 0 0 >> rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 >> binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc >> rw,nosuid,nodev,noexec,relatime 0 0 >> >> If I tried to un-mount (umount) the "ghost" mount point, it will fail with >> below error message saying that it is not found in /proc/mounts: >> >> root@hv-kvm-02:/etc# umount /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 >> /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 was not found in /proc/mounts >> /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 was not found in /proc/mounts >> >> Do you have any idea what could be the problem? Is it libvirt issue or >> CloudStack issue? >> >> Any help on this is greatly appreciated. Since I am upgrading a production >> setup, we only have 1-2 hours left to resolve the problem, otherwise we >> will be forced to revert back to 4.1.1 (for the fourth time). :( >> >> Urgently looking forward to your reply, thank you. >> >> Cheers. >> >> >> >> >>> >>> On Thu, Oct 3, 2013 at 9:53 AM, Indra Pramana <in...@sg.or.id> wrote: >>> > Dear all, >>> > >>> > We face a major problem after upgrading to 4.2.0. Mounting from KVM >>> hosts >>> > to secondary storage seems to fail, every time a new VM instance is >>> > created, it will use up the / (root) partition of the KVM hosts instead. >>> > >>> > Here is the df result: >>> > >>> > ==== >>> > root@hv-kvm-02:/home/indra# df >>> > Filesystem 1K-blocks >>> Used >>> > Available Use% Mounted on >>> > /dev/sda1 4195924 >>> > 4195924 0 100% / >>> > udev 132053356 >>> 4 >>> > 132053352 1% /dev >>> > tmpfs 52825052 >>> 440 >>> > 52824612 1% /run >>> > none 5120 >>> > 0 5120 0% /run/lock >>> > none 132062620 >>> 0 >>> > 132062620 0% /run/shm >>> > cgroup 132062620 >>> 0 >>> > 132062620 0% /sys/fs/cgroup >>> > /dev/sda6 10650544 >>> 2500424 >>> > 7609092 25% /home >>> > 10.237.11.31:/mnt/vol1/sec-storage2/template/tmpl/2/288 4195924 >>> > 4195924 0 100% /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 >>> > ==== >>> > >>> > The strange thing is that df shows that it seems to be mounted, but it's >>> > actually not mounted. If you noticed, the total capacity of the mount >>> point >>> > is exactly the same as the capacity of the / (root) partition. By right >>> it >>> > should show 7 TB instead of just 4 GB. >>> > >>> > This caused VM creation to be error due to out of disk space. This also >>> > affect the KVM operations since the / (root) partition becomes full, >>> and we >>> > can only release the space after we reboot the KVM host. >>> > >>> > Anyone experience this problem before? We are at loss on how to resolve >>> the >>> > problem. >>> > >>> > Looking forward to your reply, thank you. >>> > >>> > Cheers. >>> >> >>
