On 2/10/11 3:05 PM, adi milis wrote:
Hi,
We are using MINA in my production server. Basically, it's just a simple
chat server, with custom protocol.
Mina version is 2.0.0M6, and we dont have problem with this version so
switching to 2.0.2 isn't a priority at the moment.
Well, not switching would be a major mistake. 2.0.0-M6 is frankly buggy,
and you may face an issue every now and then...
The Server OS is Linux CentOS and the clients are Brew Device (Qualcomm),
running in internal network of the carrier (CDMA based).
As the users grow, we are suffering with some performance issues. where some
of the clients are unable to login.
FYI, each client will be assigned to unique session.
Well, this is just the way MINA is designed... One connection -> one
session.
here's our investigation steps
1. We can see the 'request packet' captured in the application log,
2. The application completes the message construction for the response to
client
3. aplication perform the final session.write(response)
4. Somehow the response packets are failed to be delivered (In the client
side, it's just stop in WAITING_STATE as no packets are delivered).
Probably because your client has disconnected. There is no way for the
TCP stack to know that a client brutally shutdown its computer. You have
to deal with idle connections on the server side.
step #2 is related with database, we find the average time required is very
acceptable (less than 100ms)
100ms ?? Whaooohhh. This is incredibly high ! I would expect a DB to be
able to process a request in less than 10ms...
Let's do some maths here :
- average response time : 100ms
- number of IoProcessor is N
- number of concurrent users is 600
so you need N = 600 * 0.1 --> 60 IoProcessor to handle the load. This is
just HUGE.
In other words, with a 100ms average processing time, you can't deliver
more than 10 req/s on a single IoProcessor, unless you added an Executor
filter in the chain.
the maximum concurent user ever recorded was 600 something when the problem
starts to occure
Concurrent users != concurrent requests. How many request per second are
you trying to process when the problem occurs ?
Also how many IoProcessor do you use, how many core do you have on your
server, what is the internal network bandwith ?
java memory configuration -Xmx1024m
ulimit -a from console
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 131072
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 71679
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
basically I'm just change the maximum open file setting for the application.
My Questions are :
1. Is there any other possible tuning required in the OS configuration
The first step is to discover where is your bottleneck. Is it the DB ?
MINA ? Your application ?
2. How to deploy scalable mina application. Let say that I want to deploy my
application in N servers.
That should be easy assuming that your users are always redirected to
the same MINA server, or if your connection is stateless.
3. Is it possible that any setting required from the carrier (DNS, Timeout,
Dormant time) may have affect the chat application ?
Difficult to say. You have to analyze the traffic to see what's going on.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
