On 14 Jan 2006, at 22:27, lichtner wrote:
On Fri, 13 Jan 2006, James Strachan wrote:
The infiniband transport would be native code, so you could use JNI.
However, it would definitely be worth it.
Agreed! I'd *love* a Java API to Infiniband! Have wanted one for ages
& google every once in a while to see if one shows up :)
It looks like MPI has support for Infiniband; would it be worth
trying to wrap that in JNI?
http://www-unix.mcs.anl.gov/mpi/
http://www-unix.mcs.anl.gov/mpi/mpich2/
I did find that HP has a Java interface for MPI. However, to me it
doesn't
necessarily seem that this is the way to go. I think for writing
distributed computations it would be the right choice, but I think
that
the people who write those choose to work in a natively compiled
language,
and I think that this may be the reason why this Java mpi doesn't
appear
to be that well-known.
However I did find something which might work for us, namely UDAPL
from the DAT Collaborative. A consortium created a spec for
interface to
anything that provides RDMA capabilities:
http://www.datcollaborative.org/udapl.html
The header files and the spec are right there.
I downloaded the only release made by infiniband.sf.net and they claim
that it only works with kernel 2.4, and that for 2.6 you have to use
openib.org. The latter claims to provide an implementation of UDAPL:
http://openib.org/doc.html
The wiki has a lot of info.
From the mailing list archive you can tell that this project has a
lot of
momentum:
http://openib.org/pipermail/openib-general/
Awesome! Thanks for all the links
I think the next thing to do would be to prove that using RDMA as
opposed
to udp is worthwhile. I think it is, because JITs are so fast now,
but I
think that before planning anything long-term I would get two
infiniband-enabled boxes and write a little prototype.
Agreed; the important issue is gonna be, can Java with JNI (or Unsafe
or one of the alternatives to JNI: http://weblog.janek.org/Archive/
2005/07/28/AlternativestoJavaNativeI.html) work with RDMA using
native ByteBuffers so that the zero copy is avoided and so that
things perform better than just using some Infiniband-optimised TCP/
IP implementation. Though to be able to test this we need to make a
prototype Java API to RDMA :) But it is definitely well worth the
experiment IMHO
The main big win is obviously avoiding the double copy of working
with TCP/IP though there are other benefits like improved flow
control (you know that you can send a message to a consumer & how
much capacity it has at any point in time so there is no need to
worry about slow/dead consumer detection) another is concurrency; in
a point-cast model, sending to multiple consumers in 1 thread is
trivial (and multi-threading definitely slows down messaging as we
found in ActiveMQ).
In ActiveMQ the use of RDMA would allow us to do straight through
processing for messages which could dramatically cut down on the
number of objects created per message & the GC overhead. (Indeed
we've been musing that it might be possible to avoid most per-message
dispatch object allocations if selectors are not used and we wrote a
custom RDMA friendly version of ActiveMQMessage; we should also be
able to optimise the use of the Journal as we can just pass around
the ByteBuffer rather than using OpenWire marshalling.
I think Appro sells
infiniband blades with Mellanox hcas.
There is also IBM's proprietary API for clustering mainframes, the
Coupling Facility:
http://www.research.ibm.com/journal/sj36-2.html
There are some amazing articles there.
Great stuff - thanks for the link!
Personally I also think there is value in implementing replication
using
udp (process groups libraries such as evs4j), so I would pursue
both at
the same time.
Yeah; like many things in distributed systems & messaging; it all
depends on what you are doing as to what solution is the best for
your scenario. Certainly both technologies are useful tools to have
in your bag when creating middleware. I personally see RDMA as a
possible faster alternative for TCP/IP inside message brokers such as
ActiveMQ as well as for request-response messaging such as in openejb.
James
-------
http://radio.weblogs.com/0112098/
___________________________________________________________
To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com
