Hi,
We are seeing an issue with stratos running out of file handles when creating a
number of applications and VM instances:
The scenario is as follows:
13 applications are deployed, each with a single cluster and a single member
instance,
As the VMs spin up stratos becomes unresponsive and checking the logs we find
the following exceptions (see below). I remember we had seen similar issues
(same exceptions) back in stratos 4.0 in the context of longevity tests.
We are running stratos 4.1 RC4 with the latest commit
commit 0fd41840fb04d92ba921bf58c59c2c3fbad0c561
Author: Imesh Gunaratne <im...@apache.org>
Date: Tue Jul 7 12:54:47 2015 +0530
Is this a known issue which might have been fixed in a later commit or
something new ? Can we verify that the fixes for the previous issues are
included in our system (jars, commit,s etc ...) ?
rg.apache.thrift.transport.TTransportException: java.net.SocketException: Too
many open files
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:106)
at
org.wso2.carbon.databridge.receiver.thrift.internal.ThriftDataReceiver$ServerThread.run(ThriftDataReceiver.java:199)
at java.lang.Thread.run(Thread.java:745)
TID: [0] [STRATOS] [2015-08-17 17:38:17,499] WARN
{org.apache.thrift.server.TThreadPoolServer} - Transport error occurred during
acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Too
many open files
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:106)
at
org.wso2.carbon.databridge.receiver.thrift.internal.ThriftDataReceiver$ServerThread.run(ThriftDataReceiver.java:199)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Too many open files
// listing the applications, member isntances and cartridge state:
[di-000-xxx] - application name,
di-000-010: applicationInstances 1, groupInstances 0, clusterInstances 1,
members 1 (Starting 1)
di-000-011: applicationInstances 1, groupInstances 0, clusterInstances 1,
members 1 (Initialized 1)
cartridge-proxy: applicationInstances 1, groupInstances 0, clusterInstances 1,
members 1 (Active 1)
di-000-001: applicationInstances 1, groupInstances 0, clusterInstances 1,
members 1 (Active 1)
di-000-002: applicationInstances 1, groupInstances 0, clusterInstances 1,
members 1 (Active 1)
di-000-012: applicationInstances 1, groupInstances 0, clusterInstances 1,
members 1 (Created 1)
di-000-003: applicationInstances 1, groupInstances 0, clusterInstances 1,
members 1 (Starting 1)
di-000-004: applicationInstances 1, groupInstances 0, clusterInstances 1,
members 1 (Starting 1)
di-000-006: applicationInstances 1, groupInstances 0, clusterInstances 1,
members 1 (Starting 1)
di-000-005: applicationInstances 1, groupInstances 0, clusterInstances 1,
members 1 (Starting 1)
di-000-008: applicationInstances 1, groupInstances 0, clusterInstances 1,
members 1 (Starting 1)
di-000-007: applicationInstances 1, groupInstances 0, clusterInstances 1,
members 1 (Starting 1)
di-000-009: applicationInstances 1, groupInstances 0, clusterInstances 1,
members 1 (Starting 1)
