We came across an interesting issue yesterday in one of our clusters. We ran
out of inodes on all of our cluster members (since when does this happen in
2012?). When this happened, it in turn made the / filesystem a read-only
filesystem which in turn made all the hosts go in to emergency maintenance mode
and as a result get marked down by Cloudstack. We found that it was caused by
hundreds of thousands of stale socket files in /tmp named
"stream-unix.####.######". To resolve the issue, we had to delete those stale
socket files (find /tmp -name "*stream*" -mtime +7 -exec rm -v {} \;), then
kill and restart xapi, then correct the emergency maintenance mode. These
hosts had only been up for 45 days before this issue occurred.
In our scouring of the interwebs, the only other instance we've been able to
find of this (or similar) happening is in the same setup we are currently
running. Xenserver 6.0.2 with CS 3.0.2. Do these stream-unix sockets have
anything to do with Cloudstack? I would think if this was a Xenserver issue
(bug), there would be a lot more on the internet about this happening. For a
temporary workaround, we've added a cronjob to cleanup these files but we'd
really like to address the actual issue that's causing these sockets to become
stale and not get cleaned-up.
Thoughts?
Thanks,
Caleb
