[ https://issues.apache.org/jira/browse/CLOUDSTACK-105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664265#comment-13664265 ]
Stephen Turner commented on CLOUDSTACK-105: ------------------------------------------- Could someone who can reproduce this attach a XenServer bugtool (aka Server Status Report from XenCenter) taken immediately after it occurs? That will enable us to track this down. Thank you. > /tmp/stream-unix.####.###### stale sockets causing inodes to run out on > Xenserver > --------------------------------------------------------------------------------- > > Key: CLOUDSTACK-105 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-105 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Third-Party Bugs > Affects Versions: pre-4.0.0 > Environment: Xenserver 6.0.2 > Cloudstack 3.0.2 > Reporter: Caleb Call > Assignee: Devdeep Singh > Fix For: 4.1.0 > > Attachments: messages > > > We came across an interesting issue in one of our clusters. We ran out of > inodes on all of our cluster members (since when does this happen in 2012?). > When this happened, it in turn made the / filesystem a read-only filesystem > which in turn made all the hosts go in to emergency maintenance mode and as a > result get marked down by Cloudstack. We found that it was caused by > hundreds of thousands of stale socket files in /tmp named > "stream-unix.####.######". To resolve the issue, we had to delete those > stale socket files (find /tmp -name "*stream*" -mtime +7 -exec rm -v {} \;), > then kill and restart xapi, then correct the emergency maintenance mode. > These hosts had only been up for 45 days before this issue occurred. > In our scouring of the interwebs, the only other instance we've been able to > find of this (or similar) happening is in the same setup we are currently > running. Xenserver 6.0.2 with CS 3.0.2. Do these stream-unix sockets have > anything to do with Cloudstack? I would think if this was a Xenserver issue > (bug), there would be a lot more on the internet about this happening. For a > temporary workaround, we've added a cronjob to cleanup these files but we'd > really like to address the actual issue that's causing these sockets to > become stale and not get cleaned-up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira