[
https://issues.apache.org/jira/browse/CLOUDSTACK-105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510950#comment-13510950
]
Jason Bausewein edited comment on CLOUDSTACK-105 at 12/6/12 12:07 AM:
----------------------------------------------------------------------
I am able to reproduce this issue consistently. I have a single xen host in a
basic zone. I tracked it down to the following process creating the
stream-unix.****.**** files about every 10 seconds.
root 8237 8223 0 09:59 ? 00:00:00 ovs-vsctl add-br xapi0
Dec 5 09:59:15 xenserver1 ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl
add-br xapi0
Dec 5 09:59:15 xenserver1 ovs-vsctl:
00002|stream_unix|ERR|/tmp/stream-unix.8237.0: connection to
/var/run/openvswitch/db.sock failed: No such file or directory
Dec 5 09:59:15 xenserver1 ovs-vsctl:
00003|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt
failed (No such file or directory)
Dec 5 09:59:16 xenserver1 ovs-vsctl:
00004|stream_unix|ERR|/tmp/stream-unix.8237.1: connection to
/var/run/openvswitch/db.sock failed: No such file or directory
Dec 5 09:59:16 xenserver1 ovs-vsctl:
00005|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt
failed (No such file or directory)
Dec 5 09:59:18 xenserver1 ovs-vsctl:
00006|stream_unix|ERR|/tmp/stream-unix.8237.2: connection to
/var/run/openvswitch/db.sock failed: No such file or directory
Dec 5 09:59:18 xenserver1 ovs-vsctl:
00007|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt
failed (No such file or directory)
These messages start immediately on boot. I attached my messages log file.
I am not using openvswitch. Is there something I need to turn off?
was (Author: jbausewein):
I am able to reproduce this issue consistently. I have a single xen host
in a basic zone. I tracked it down to the following process creating the
stream-unix.****.**** files about every 10 seconds.
root 8237 8223 0 09:59 ? 00:00:00 ovs-vsctl add-br xapi0
Dec 5 09:59:15 xenserver1 ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl
add-br xapi0
Dec 5 09:59:15 xenserver1 ovs-vsctl:
00002|stream_unix|ERR|/tmp/stream-unix.8237.0: connection to
/var/run/openvswitch/db.sock failed: No such file or directory
Dec 5 09:59:15 xenserver1 ovs-vsctl:
00003|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt
failed (No such file or directory)
Dec 5 09:59:16 xenserver1 ovs-vsctl:
00004|stream_unix|ERR|/tmp/stream-unix.8237.1: connection to
/var/run/openvswitch/db.sock failed: No such file or directory
Dec 5 09:59:16 xenserver1 ovs-vsctl:
00005|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt
failed (No such file or directory)
Dec 5 09:59:18 xenserver1 ovs-vsctl:
00006|stream_unix|ERR|/tmp/stream-unix.8237.2: connection to
/var/run/openvswitch/db.sock failed: No such file or directory
Dec 5 09:59:18 xenserver1 ovs-vsctl:
00007|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt
failed (No such file or directory)
I am not using openvswitch. Is there something I need to turn off?
> /tmp/stream-unix.####.###### stale sockets causing inodes to run out on
> Xenserver
> ---------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-105
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-105
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: XenServer
> Affects Versions: pre-4.0.0
> Environment: Xenserver 6.0.2
> Cloudstack 3.0.2
> Reporter: Caleb Call
> Assignee: Devdeep Singh
> Fix For: 4.1.0
>
> Attachments: messages
>
>
> We came across an interesting issue in one of our clusters. We ran out of
> inodes on all of our cluster members (since when does this happen in 2012?).
> When this happened, it in turn made the / filesystem a read-only filesystem
> which in turn made all the hosts go in to emergency maintenance mode and as a
> result get marked down by Cloudstack. We found that it was caused by
> hundreds of thousands of stale socket files in /tmp named
> "stream-unix.####.######". To resolve the issue, we had to delete those
> stale socket files (find /tmp -name "*stream*" -mtime +7 -exec rm -v {} \;),
> then kill and restart xapi, then correct the emergency maintenance mode.
> These hosts had only been up for 45 days before this issue occurred.
> In our scouring of the interwebs, the only other instance we've been able to
> find of this (or similar) happening is in the same setup we are currently
> running. Xenserver 6.0.2 with CS 3.0.2. Do these stream-unix sockets have
> anything to do with Cloudstack? I would think if this was a Xenserver issue
> (bug), there would be a lot more on the internet about this happening. For a
> temporary workaround, we've added a cronjob to cleanup these files but we'd
> really like to address the actual issue that's causing these sockets to
> become stale and not get cleaned-up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
