On Wed, 2010-10-06 at 16:55 +0200, Emmanuel Lecharny wrote: > On 10/6/10 2:49 PM, Oleg Kalnichevski wrote: > > MINA devs, > > > > Would you be receptive to the idea of supporting a message-less (or > > bare-metal, if you like) I/O mode? Essentially, I would like MINA 3.0 to > > make it possible to interact directly with the underlying I/O channels > > bypassing the read / write data queues altogether. > There is nothing such as a queue used for read operation. We read the > data from the channel into a buffer, and then call the messageReceived() > method through the filters chain up to he handler. > > Of course, if you add an executor in the middle, then that's a different > story. > > Anyway. Maybe what you want is that the handler directly do the read, > sparing the creation of an intermediate buffer. You can't currently do > that, so the buffer will always be created and filled with data. >
There are enough scenarios (especially when streaming large entities) when this extra copy is wasteful and unnecessary. Moreover, what if the I/O handler is simply unable to consume the message entirely without allocating more memory? The content of the message cannot be unread until some space frees up in the input buffer. The beauty of Channel is that the I/O handler can read only as much as it is able to process at a given point of time. > For the write operation, what is currently missing in MINA 2 is the > transfertTo method which would allow you to push file contents directly > to the socket without copying it into memory. This is most certainly > something we want to have in 3.0 > Again, what if the transmitted entity is not backed by a file? > > In other words, > > whenever a channel signals read / write readiness, the I/O selector > > thread would fire an event referring to the originating channel. The > > decision as to how much data can be read from or written to the channel > > would be left up to the I/O handler. > > Well, that does not seems very useful. It all depends how you look at it. > If you defer to the handler the > read and write operation on top of the channel, then why not directly > write your NIO server from scratch ? It's about 500 lines of code, all > in all, and you can even disassemble MINA to do that ... It is a bit more if the SSL support and other bits and pieces, but this is exactly what I have to do at the moment. > > I am perfectly aware of downsides of this approach, which are several, > > but it would enable data intensive protocols such as HTTP and SMTP to > > manage connection memory footprint more conservatively. For instance, > > the protocol handler could attempt to preallocate a fixed and invariable > > amount memory at the connection initialization time and either succeed > > or fail early instead of risking the out of memory condition half way > > through a transaction due to the memory flooding. > You already can manage the flooding by limitating the number of bytes > you read. As you have the complete control on the initial RcvBufferSize, > plus the control over the created buffer, you can always at some point > 'kill' a session which has receive too much data. > I would very much rather prefer to avoid memory flooding instead of having to manage it. > There is also possible to manage a threshold for every session, which > might help in this case. > > For instance, in Apache DirectoryServer, we have added a parameter that > limit the size of the received PDU, this parameter can be modified for > any session. If we receive more bytes that this limit, then we close the > session. It works pretty well. By dropping the connection in the middle of the session? Would not it be better to pre-allocate a fixed amount of memory and have a guarantee that more memory will not be required (at least on the transport level)? > > I understand this might be too much of a radical shift from the existing > > architecture, so feel free to ignore me if you find this approach > > incompatible with the MINA 3.0 design concepts. > No, that's fine, it's just important to know what we already have and > what we can offer without cutting two legs to MINA :) > It is shame that by abstracting away java.nio.channels.Channel both MINA and Netty throw away one of the best features of NIO, which is memory efficiency. Anyways, I understand that MINA is based on a different philosophy and I also understand the convenience of having a chain of filters that work with messages that are essentially small chunks of memory. I just thought the core framework might be able to support both modes. I apologize for intrusion. Cheers Oleg
