On 4/25/11 11:46 PM, Semih Salihoglu wrote:
Hi All,
I am new to MINA and am trying to implement the communication layer of a
distributed system in which all my machines (I have 9 of them) send to each
other similar amounts of data. By that I mean, they send each other same
type of messages and in similar volumes. In the current code that I have for
example, each machine sends to each other machine roughly 8MB of data. That
is each machine sends a total of 64 MB of data to other machines and at the
same time receives 64 MB of data from other machines.
I have been trying to do this efficiently with java io over the last few
weeks and I couldn't and now I want to switch to MINA. Before this decision
here are the numbers I was getting: All of my machines are connected to each
other directly by 1Gbit ethernet cable. Theoretically if there are only 2
machines, reading/writing 64MB of data should take around 500 millis (I
wrote a simple code to just send and receive this much data between 2
machines and in java I get 600millis, which is very close to the theoretical
number).
Your theorical number is just worng. The time you've got writing the
code is right. Sending/receiving 64Mb, on a Gb ethernet network will
takes more than 500 ms, as you have some overhead plus potential
collision. 600ms is actually quite a good number.
However with the overhead of parsing the data on the receiving
side, the number goes up roughly to 1.5seconds: there is some lock
acquiring, some object allocations, etc.. However when I go from 2 machines
to 9 machines, the number goes from 1.5 seconds to 8 seconds. I have been
trying to nail down exactly where the bottleneck is and although I can't say
I did a very good job, so far I have noticed that a) the receiving of the
messages is in general slow and b) when 8 machines are concurrently writing
to a single machine, their write performance goes down significantly (in
blocking io, sending 8MB of data should take<100 millis, but it takes close
to 1 second. this time is measured on the sender side).
That's expected.
Back to MINA: I wrote a simple code to replicate this scenario and here's
what I did:
- In the beginning of my code, all my machines establish connections with
each other (if machine i wants to connect to machine j, and i< j, then i is
the client and j is the server but once the session is established both
machines read/write on the same IoSession object). I store the IoSession
object for each machine in a map. I look at the time when all machines have
established connections.
- I then wrote a simple computation thread on each machine and it starts
generating dummy messages for each other machine and puts the messages into
outgoing IoBuffers. When an outgoing IoBuffer is full, the computation
thread wraps them into a simple object called SimpleMessage and writes it to
the IoSession object for that machine. The IoBuffers are 1MB each. I will
loop 8 times so a total of 8MB will be sent to each machine.
- I wrote a MessageEncoder object which simply writes the wrapped
IoBuffer to the ProtocolEncoderOutput.
- For receiving data there is a MessageDecoder that extends
CumulativeProtocolDecoder. This decoder keeps skipping over the available
data in the IoBuffer keeping track of how much of the 8MB it has skipped.
When it is done skipping the entire 8MB of data, it says it's done. And I
look at System.currentTimeMillis when this happens.
- I have a dummy MessageHandler that doesn't do anything. It just logs
the events and I see that each time my computation thread writes to a
IoSession object, I get a MessageSent event as expected.
I haven't yet optimized MINA, I am not that familiar with the parameters
yet. But the numbers I have been getting has been similar to the numbers
with regular io (again for 64MB of data about 1.5 seconds between 2
machines, 7-8 seconds between 9 machines, but without any parsing). I am
using Mina 2.0.3. My questions are
1. Is there a major flaw or inefficiency in this kind of design given my
communication pattern?
You are simply overloading your network, when all the machines are
talking to each others. It generates many collisions on the network. You
are trying to transmit 64Mb to 8 machines, which will simply take 8
times as long as it cost to send 64Mb between twO machines.
2. I have noticed that my decoder is being called almost always in every
64KB. Is there a hidden 64KB buffer somewhere that I can maybe make bigger
to see whether or not it affects the performance.
You can select a bigger buffer for the connection. However, it won't
improve a lot the transmission. Seems like you are network bounded here.
Also I realized that 64MB
Are you kidding ? Again, 64Mb will saturate your Gb ethernet during 600
ms...
is very small but this is just for testing. Yet I
think it should take much less to send this much data across machines but
for some reason I have failed to find the right implementation.
Install more network cards in your machines.Don't put all of them on the
same Gb ethernet networrk.
And don't send that much data...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
