Re: Many message decoder/encoder
Hi, If you have lots of different types of binary messages , check out
my litle library javastruct (http://code.google.com/p/javastruct/)
There are some examples and documents as well
http://code.google.com/p/javastruct/wiki/HowToUseJavaStruct
http://code.google.com/p/javastruct/wiki/example_photoshop_acb_file_reader_writer
and an incomplete example for mina
http://javastruct.googlecode.com/svn/trunk/javastruct/samples/mina/
if your protocol is not binary, you can ignore all this. I saved a lot
of time and code using this approach (we had more than 50 binary
little endian messages)
Mehmet
On Nov 9, 2007 10:22 AM, csantini <[EMAIL PROTECTED]> wrote:
>
> I have many Message classes that implements MessageDecoder and
> MessageEncoder, and they are all listed in a DemuxingProtocolCodecFactory.
> Now I need to be able to choose what decoder/encoder to use just depending
> on the second byte (opcode) I receive; I could implement a decodable() for
> each Message returning OK if the byte is right but it's slow with many
> Messages! I need to do something like this:
>
> switch(opCodeByte){
> case 0x01: use HelloMessageDecoder
> case 0x02: use DataMessageDecoder
> ...
> }
>
> Maybe DemuxingIoHandler is the right way but there is no doc about, could
> you help me ?
> Thanks for your work and your beautiful framework!
>
> --
> View this message in context:
> http://www.nabble.com/Many-message-decoder-encoder-tf4776815s16868.html#a13664414
> Sent from the Apache MINA Support Forum mailing list archive at Nabble.com.
>
>
Re: MINA's scalability
On 10/27/07, Bogdan Ciprian Pistol <[EMAIL PROTECTED]> wrote: > Hi Gato, > > > You should be just fine using MINA. We are using MINA in Openfire and so > > far we got up to 170K concurrent connections in a single JVM. > > Thank you for responding so quickly, > > Because MINA can handle 170K connections (which is more than enough > for me) I will use MINA for my project. I'm glad I found this project, > it seems that this is the only serious network toolkit for Java I > found so far. Do you know if there are any alternatives to this > toolkit?, just to have a broad view before starting to work on my > project. > > Bogdan > Yes Mina is perfect, There is also project Grizzly https://grizzly.dev.java.net/ Don't make your own :)
Re: New name for ByteBuffer?
I say DataBuffer or MinaByteBuffer On 9/18/07, Trustin Lee <[EMAIL PROTECTED]> wrote: > Hi folks, > > It is often confusing to discriminate MINA ByteBuffer and NIO > ByteBuffer. Do we need renaming? I didn't have much difficulties > actually because most Java code doesn't use both types at the same > time. > > There was an opinion about renaming it to MinaByteBuffer, but I don't > think it's the best name available for us. I think DataBuffer, > ExtendedByteBuffer, ExtendedBuffer or just Buffer might also be a > candidate. There's Buffer in NIO, too, but nobody uses that class > directly. > > I'd like to find the best name; short and not confusing one. Please > don't hesitate to respond to this message with your idea so we can > find out the best alternative. > > Trustin > -- > what we call human nature is actually human habit > -- > http://gleamynode.net/ > -- > PGP Key ID: 0x0255ECA6 >
Re: Is MINA the right library?
On 9/12/07, Sajit Zachariah <[EMAIL PROTECTED]> wrote: > Hi All, > > I am sure that this question is not very appropriate > in this mailing list. > > But could you suggest if MINA is the right API for the > application that we are writing for the following > requirements. > > 1. application should manage multiple devices on > tcp/ip > 2. application should be able to listen to the alarms > and events from multiple devices > 3. application should be able send commands to the > devices; these devices could be on LAN or on internet > > Thanks for all your inputs > > -Sajit > I have implemented a similar server using Mina, which was capable of, - managing multiple devices (~500 - 1000 devices, all on-line) - lots of different binary message types - listening alarms and other data from devices - sending messages to devices Mina did the job perfectly. Mehmet.
Re: Mina throughput
Hi mat, Actually it is not faster than any hand made protocol encoder - decoder, since it has the overhead of reflection. But most of the time the problem is not the encoding and decoding of streams, it is usually I/O or processing (not decoding or encoding) of messages that causes the performance problems. The aim of library is to make life easier if you have lots of different types of messages to encode and decode. But it is fast enough for most applications. It can encode or decode more than 100.000 simple and small messages per second on an average box. You should try it yourself to see whether it is appropriate for your application. As for the endian problem, it also contains Little endian data streams for the job. There is also a mina example in the samples directory.. It is not complete, but gives an idea for use of javastruct for network protocols. Mehmet On 8/21/07, mat <[EMAIL PROTECTED]> wrote: > Thanks. I remember Trustin mentioned your project once in this forum. I did > some research last night before I saw your thread. :) BTW, how do you handle > Big Little Endian thing in your project? The examples and tutorials are too > simple to understand. and why you consider it is fast? > > On 8/22/07, Mehmet D. AKIN <[EMAIL PROTECTED]> wrote: > > > > On 8/21/07, mat <[EMAIL PROTECTED]> wrote: > > > The reason why I raised this thread is: recently I am really bothered > > with > > > following issue. > > > > > > My mina component has to connect to a legacy server written in C++. C++ > > > server just simply sends out the struct as a message. I think for a C++ > > > program it is just simple return the struct pointer and use each field. > > > However, to my java program that is a disaster(also really lower the > > > performance). I have to use all the system.arraycopy to make up each > > > field.(Actually it causes OOM). I don't know if you guys ever faced this > > > kind of problem. How to solve it? > > > > > [snip] > > > > well there is a small library I am working on, It helps protocol > > programmers by allowing java classes to be somewhat treated as C - C++ > > structs. You might want to check it, it uses annotations and > > reflection and reasonably fast. > > I am still working on it so there is no guarantee that its interfaces > > wont change, but it works, and there are examples on the site (an > > adobe color book reader-writer), check the wiki pages of project. > > > > site is: http://code.google.com/p/javastruct/ > > > > Mehmet > > >
Re: Mina throughput
On 8/21/07, mat <[EMAIL PROTECTED]> wrote: > The reason why I raised this thread is: recently I am really bothered with > following issue. > > My mina component has to connect to a legacy server written in C++. C++ > server just simply sends out the struct as a message. I think for a C++ > program it is just simple return the struct pointer and use each field. > However, to my java program that is a disaster(also really lower the > performance). I have to use all the system.arraycopy to make up each > field.(Actually it causes OOM). I don't know if you guys ever faced this > kind of problem. How to solve it? > [snip] well there is a small library I am working on, It helps protocol programmers by allowing java classes to be somewhat treated as C - C++ structs. You might want to check it, it uses annotations and reflection and reasonably fast. I am still working on it so there is no guarantee that its interfaces wont change, but it works, and there are examples on the site (an adobe color book reader-writer), check the wiki pages of project. site is: http://code.google.com/p/javastruct/ Mehmet
Re: How to get MINA to detect a lost connection?
AFAIK the only reliable way to detect the loss of connection is having some sort of keep-alive or heartbeat mechanism for socket communications. On 8/15/07, Yigal Rachman <[EMAIL PROTECTED]> wrote: > > Hi, Mat: > > Thanks for your amazingly quick response! > > DIRMINA-354 appears to refer to the issue of a "keep-alive" message, which > is great if both sides can cooperate on this. My problem is that the > instrument uses a proprietory third-party protocol, over which I have no > control. The instrument expects to be left alone while it is doing its job. > > I was hoping that MINA has some way to detect the loss of the connection, > as happens with "normal" socket programming. > > Yigal > > > > mat wrote: > https://issues.apache.org/jira/browse/DIRMINA-354 > > On 8/16/07, Yigal Rachman <[EMAIL PROTECTED]> wrote: > > > Hi, Folks: > > I have an instrument driver that is built on MINA. During normal > operation, it does not write to the instrument, but just listens for > input from the instrument. This part works flawlessly with very little > coding on my part (thanks to MINA). > > There is one glitch: if the socket connection is ever lost, the driver > continues to think it is connected because MINA does not close the > session. > > Any ideas how I can get this to work properly? > > Thanks in advance, > Yigal Rachman > > DMAS Data Acquisition Developer > NEPTUNE Canada > University of Victoria, > Victoria, BC, Canada > www.neptunecanada.ca > > > > > > > > >
Re: Demuxing Tutorials
On 7/24/07, Trustin Lee <[EMAIL PROTECTED]> wrote:
Hi Rodrigo,
On 7/24/07, Rodrigo Madera <[EMAIL PROTECTED]> wrote:
> Please tell me this wheel is invented and ready in MINA!!
>
> Here is the scenario taking as an example the SumUp server (the closest
> thing to the case).
> Let's suppose there is this project called MegaMath, which is kind of like
> SumUp but has more operations:
>
> public class MegaMathMessage {
> private byte opcode;
>
> MegaMathMessage(opcode) {
> this.opcode = opcode;
> }
>
> ...
> }
>
> public class MultiplyMessage extends MegaMathMessage {
> private byte operand0;
>
> MultiplyMessage()
> {
> super(Constants.MULTIPLY_OPCODE);
> ...
> }
>
> ...
> }
>
> public class DoSomethingReallyNiceWithThreeArguments {
> private int arg0;
> private short arg1;
> private byte arg3;
>
> // ... initialize with ctor etc...
> }
>
> Now that we have the variable parameter quantity messages, we could write
> the decoders and encoders, but that's not nice.
>
> So the part we are writing let's us do basically this:
>
> NiceEngine.register(MultiplyMessage.class);
> NiceEngine.register(SingASongMessage.class);
> NiceEngine.register(DanceMessage.class);
>
> MessageEncoder multiplyMessageDecoder =
> NiceEngine.getDecoder("MultiplyMessage");
> // classname or class can be used here.
> MessageDecoder multiplyMessageEncoder = Nice... you get the point...
>
> Is this available?
It's not available. I think it's difficult to generalize.
> The problem is that manually creating Decoders and Encoders is just a waste
> of time. This should be a machine's work, and this is what our design is
> promissing to aid us in.
>
> We tried using Javastruct, and even devoted three full days to writing code
> for it, but we came to the conclusion that we needed more and in a more
> structured manner, so we started this solution.
>
> If this is already implemented, please share some light on how to use it.
You could forward your feedback about JavaStruct to Mehmet so he can
evolve it on and on. :)
Thanks,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Hi,
I agree with Rodrigo on writing protocol encoders and decoders is
quite tedious job. That was the reason I was thinking something like
JavaStruct might help on these situations. He also helped me to find
some bugs and had some nice insight on the architecture. Thanks a lot.
Although Rodrigo and his team found JavaStruct is not enough to handle
their problems, I still have a feeling that JavaStruct can be helpful
on many situations. Please look at the example codes below. There are
two examples, an Adobe Photoshop color book binary file reader -
writer and a Mina example. Mina example is based on the nice protocol
encoder-decoder tutorial on the wiki and it is not complete yet, but
if you look at the code you will get the idea.
Examples: http://javastruct.googlecode.com/svn/trunk/javastruct/samples/
I also updated how-to documnent in the wiki. there is also usable 0.1
jar in the downloads section.
IMHO, Rodrigo's approach, (If I have gueesed correctly), automatic
generation of encoders and decoders have good and weak sides. It
probably generates code with better I/O performance (since for every
type of message, there will be an encoder-decoder generated
automatically, reading and writing directly to mina byte buffers. But
generating code has always drawbacks, difficult to maintain and
implement and sometimes considered as an anti pattern. I hope they
achieve good results and share their results with us.
regards
Mehmet
Re: Interesting project for a protocol codec: JavaStruct
Ah, yes you are right, it is Javolution indeed. Thanks for the correction.
On 7/18/07, 向秦贤 <[EMAIL PROTECTED]> wrote:
Hi Mehmet,
Great!!
BTW, as follow,
Regards,
2007/7/17, Mehmet D. AKIN <[EMAIL PROTECTED]>:
>
> On 7/17/07, Trustin Lee <[EMAIL PROTECTED]> wrote:
> > On 7/17/07, Rodrigo Madera <[EMAIL PROTECTED]> wrote:
> > > I had to develop an interface with save() and load() methods to
> serialize,
> > > and that bloats my objects and impacts maintainability.
> > >
> > > This Struct project really pays off for binary protocols.
> > >
> > > Trustin,
> > > What do you have in mind? Merge? Dependency?
> >
> > The author is a long-time user of MINA, and he'd like to donate the
> > code once the project becomes more mature because it's far from
> > perfection for now. I'd like to know what people including committers
> > think about the project. If the community's response is positive, we
> > could bring the project into our sandbox and merge it into trunk when
> > it's ready.
> >
> > Trustin
> > --
> > what we call human nature is actually human habit
> > --
> > http://gleamynode.net/
> > --
> > PGP Key ID: 0x0255ECA6
> >
>
> Thanks for posting and comments Trustin and Mina users,
>
> I am the author of the javastruct. I have developed this library 3-4
> years ago (with some help of a friend), and It really helped us to
> implement complex binary communication protocols with other devices
> and applications written in C, C++. We had 70-80 different message
> types and writing a different encoding-decoding code for each would be
> very difficult and error-prone, so we decided to automatize the
> process. It is a very simple idea and code is not complex, quite
> straightforward.
>
> It was for java 1.4 and had some other extras, but I decided to start
> JavaStruct project and rewrote-simplfy most of the code and made it
> use only Java 5 annotations. it still is nowhere near finished but
> basic stuff works.
>
> Currently it supports primitives, arrays, nested Structs and Struct
> arrays. (both Little and Big endian) I also added a special kind of
> field that gives the length of another array field in the struct, this
> pattern is used a lot in C world. Something like:
> struct Foo {
> int length;
> char* buffer;}
> The length of the buffer is defined in length field. If I start to use
> Mina Byte Buffers it will be great to add alignment support and get
> rid of Litle endian Data Streams.
>
> Performancewise for Classes with 2-3 fields or a few primitive
> arrays, JavaStruct is quite faster than Java serialization, for 8-10
> primitive fields or simple arrays it is on par with java
> serialization. Please note that if you write your own decoding and
> encoding routines, they would be faster than Javastruct, because of
> reflection and sanity checks library uses. But it is still quite fast,
> 200-300 thousands simple messages per second is more than enough for
> most applications, generally bottleneck is somewhere else. I'll post
> more performance data later.
>
> Actually this is not the first java - struct library. Jean-Marie
> Dautelle's JScience library has an excellent struct Class,
BTW,
It's not JScience, but Javolution project.
but his
> approach is different then mine. mines are simple Pojo's in JScience
> you have to use special classes, but his work is more complete than
> mine.
>
> Trustin, I wish to work on this project, but it is better to make it
> mature to be somewhat usable with Mina. So lets wait while I make some
> more progress on GoogleCode project site, and prepare usable jars, so
> that mina community can test it. Is it okay If I post occasional
> progress reports on mina mailing lists?
>
> Another question is the license, currently it is MPL, if it's
> inappropriate I an change it.
>
> regards
> Mehmet D. Akın
>
--
向秦贤
Re: Interesting project for a protocol codec: JavaStruct
On 7/17/07, Trustin Lee <[EMAIL PROTECTED]> wrote:
On 7/17/07, Rodrigo Madera <[EMAIL PROTECTED]> wrote:
> I had to develop an interface with save() and load() methods to serialize,
> and that bloats my objects and impacts maintainability.
>
> This Struct project really pays off for binary protocols.
>
> Trustin,
> What do you have in mind? Merge? Dependency?
The author is a long-time user of MINA, and he'd like to donate the
code once the project becomes more mature because it's far from
perfection for now. I'd like to know what people including committers
think about the project. If the community's response is positive, we
could bring the project into our sandbox and merge it into trunk when
it's ready.
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Thanks for posting and comments Trustin and Mina users,
I am the author of the javastruct. I have developed this library 3-4
years ago (with some help of a friend), and It really helped us to
implement complex binary communication protocols with other devices
and applications written in C, C++. We had 70-80 different message
types and writing a different encoding-decoding code for each would be
very difficult and error-prone, so we decided to automatize the
process. It is a very simple idea and code is not complex, quite
straightforward.
It was for java 1.4 and had some other extras, but I decided to start
JavaStruct project and rewrote-simplfy most of the code and made it
use only Java 5 annotations. it still is nowhere near finished but
basic stuff works.
Currently it supports primitives, arrays, nested Structs and Struct
arrays. (both Little and Big endian) I also added a special kind of
field that gives the length of another array field in the struct, this
pattern is used a lot in C world. Something like:
struct Foo {
int length;
char* buffer;}
The length of the buffer is defined in length field. If I start to use
Mina Byte Buffers it will be great to add alignment support and get
rid of Litle endian Data Streams.
Performancewise for Classes with 2-3 fields or a few primitive
arrays, JavaStruct is quite faster than Java serialization, for 8-10
primitive fields or simple arrays it is on par with java
serialization. Please note that if you write your own decoding and
encoding routines, they would be faster than Javastruct, because of
reflection and sanity checks library uses. But it is still quite fast,
200-300 thousands simple messages per second is more than enough for
most applications, generally bottleneck is somewhere else. I'll post
more performance data later.
Actually this is not the first java - struct library. Jean-Marie
Dautelle's JScience library has an excellent struct Class, but his
approach is different then mine. mines are simple Pojo's in JScience
you have to use special classes, but his work is more complete than
mine.
Trustin, I wish to work on this project, but it is better to make it
mature to be somewhat usable with Mina. So lets wait while I make some
more progress on GoogleCode project site, and prepare usable jars, so
that mina community can test it. Is it okay If I post occasional
progress reports on mina mailing lists?
Another question is the license, currently it is MPL, if it's
inappropriate I an change it.
regards
Mehmet D. Akın
Re: grizzly versus mina
On 5/24/07, Adam Fisk <[EMAIL PROTECTED]> wrote: The slides were just posted from this Java One session claiming Grizzly blows MINA away performance-wise, and I'm just curious as to people's views on it. They present some interesting ideas about optimizing selector threading and ByteBuffer use. http://developers.sun.com/learning/javaoneonline/j1sessn.jsp?sessn=TS-2992&yr=2007&track=5 Maybe someone could comment on the performance improvements in MINA 2.0? It might also be useful to look at Grizzlies techniques to see if MINA could incorporate them. I know at least Scott Oaks from Grizzly is a solid performance guy, so their numbers are likely correct. Quick note: I'm not trying to spark a Grizzly/MINA battle by any means. I just started using MINA after having implemented several generic NIO frameworks myself, and I absolutely love MINA's approach. It allowed me to code a STUN server in 2 days, and I'm porting my SIP server now. Thanks, Adam Still benchmarks presented in that document seems contradicting Trustin's benchmarks. In the presentation it says Grizzly with Async web is just a little faster than a "C based web server " but almost two times faster and scalable than Mina.But Trustin's test also shows that Mina based Async Web server is a little faster than a C based Server (Apache HTTP) so there is something fishy going on here.. If only we had a wider benchmark suite which tests for different types of protocols and loads, instead of microbenchmarks. Mehmet
Re: [VOTE] Release MINA 1.1.0
[X]: +1, release Have been testing for a while..
Re: Separating MINA ByteBuffer classes into a new JAR package.
Mina is already a tiny package with a lot of power in it, I dont see the point to extract a vital part of it as a seperate jar. Distributing, using and managing a single jar is always better and easier. I know its not a big deal, but to me simpler is better. in short, I am totally agree with Jeroen Brattinga. A seperate automatically generated ByteBuffer.jar is ok as long as it is also in mina. On 2/20/07, Alex Karasulu <[EMAIL PROTECTED]> wrote: Without sounding like my 5 year old :), but why not? Could you give more reasons than just your opinion? Alex On 2/19/07, Mehmet D. AKIN <[EMAIL PROTECTED]> wrote: > > On 1/21/07, Trustin Lee <[EMAIL PROTECTED]> wrote: > > Hi, > > > > The MINA ByteBuffer classes can be used without MINA for easier > manipulation > > of NIO ByteBuffers. For now, they belong to mina-core JAR, but we could > > split it even further to the two JARs; mina-bytebuffer, mina-core, and > > mina-transport-nio. By this sepration, we could remove some bad cyclic > > dependencies between transport implementation and the core API and allow > > users enjoy the access to the powerful byte buffer manipulation without > > using MINA. > > > > WDYT? > > -1 > > I dont think mina should be split even further. Maybe they could be > seperated for external use but Mina core should contain them too. >
Re: Separating MINA ByteBuffer classes into a new JAR package.
On 1/21/07, Trustin Lee <[EMAIL PROTECTED]> wrote: Hi, The MINA ByteBuffer classes can be used without MINA for easier manipulation of NIO ByteBuffers. For now, they belong to mina-core JAR, but we could split it even further to the two JARs; mina-bytebuffer, mina-core, and mina-transport-nio. By this sepration, we could remove some bad cyclic dependencies between transport implementation and the core API and allow users enjoy the access to the powerful byte buffer manipulation without using MINA. WDYT? -1 I dont think mina should be split even further. Maybe they could be seperated for external use but Mina core should contain them too.
Some questions (slf4j , how to reduce thread count, accessing session IP address)
Hi, I have three questions (Using mina 1.1 pre release snapshot) 1. I failed to find a single slf4j jar file in sl4j1.1 distribution to satisfy mina's dependencies. I had to include both slf4j-api and slf4j-nop jars to run tests. Is there a single slf4j jar to run mina? 2. As far as I see, MINA automatically creates 16 threads as more than one client tries to access to server. Is there a way to reduce this thread count.. to like 1 or 2 ? I could do this in an old version but API seems to be changed. 3. I want to access the connected clients IP address from session info. again I could do this, but not anymore :) What I want to do is to allow "only local connections" so I will check the address on connection request. There is a blacklist filter for such purposes but, I guess a whitelist filter approach is needed in this situation (allow this, deny all others). I browsed the documentations but either I couldnt see the answers or API's are changed since the documentation is written. regards Mehmet
Java 5 jars
I have seen a long thread about full java 5 support for Mina in 1.2 version. Is there any test jars for Java 5 support and no backport library dependency, like Mina 1.1 ? regards. Mehmet
Re: A minimal jar?
.. snip .. Is there any reason that you want to cut out 250K? Are you using MINA in an embedded environment? Trustin Actually we are using netty on a mini application working as a spelling checker server for all desktop applications. I want to cut out all unnecessary stuff to achieve smaller memory footprint and faster start-up, since it is essential for the desktop system. Mehmet.
A minimal jar?
I once used netty in a project and it was a 80-90K jar, Mina 1.0 is like 350K. Is there a minimal deployment option that provides simple socket server functionality but yet smaller than 100K?
