> > I thought about adding setOrder() method and ended up with not adding it > because it can cause thread-safety issue when the queue is used in > consumer-producer scenario. For example, two threads could change the order > at the same time and then the order can be mixed up. Moreover, byte order > doesn't change often, so I thought it might be a good idea to let user > specify it when the queue is constructed.
Good point, this will be ok. > For now, slicing might copy or might not copy depending on the specified > position and length. If the requested slice belongs to one ByteBuffer, no > copy will be performed. Otherwise a new ByteBuffer will be created and > zero-copy in not realized. Decoder implementor needs to understand this and > implement the decoder properly to achieve zero-copy. State machine based > decoders should not have a problem with this IMHO. Please let me know if I > am missing something. What about slicing ByteBufferQueues to another ByteBufferQueue by default while reusing (e.g. duplicate() while using the same underlying byte array) the affected underlying buffers...with the ability to transform the contents to a single bytebuffer for backward compatibility (queue.toByteBuffer())? When doing this, the default behaviour will be zero-copy and ByteBuffers will still be used. I think mixing ByteBufferQueues with ByteBuffers by the default implementation is not a perfect behaviour. If I slice a Queue, I expect a Queue to be returned in the first place. > To provide more perfect zero-copy, we need to introduce a new type which > wraps one or more ByteBuffers. Its implementation could be similar to > ByteBufferQueue, but it's different in that its length is fixed and it looks > exactly like ByteBuffer. It's because ByteBuffer cannot be extended. > Should we provide this? I think we can. The problem is that we need to > write a whole bunch of JavaDoc for the new buffer class, although it's just > one time task. Yes, this would be the first and imho the best possibility to achive this behaviour. But do we really need some huge wrapper class? Afaik e.g. HeapByteBuffer is already a wrapper for the relevant byte array. What about just duplicating bytebuffers (as it keeps the same underlying byte array, see constructors inside ByteBuffer sources) for every queue instance so that it gains its own position / limit / etc. values but still using the same underlying byte array? Isn't this basically the same? Of course, when manipulating one bytebuffer's contents, the other's will be modified too - but I don't see any problem that may occur because of this because the content is modified mostly on creation and when explicitly reusing buffers only and slicing is mostly done when reading. However, this only makes sence when the initial position and limit of a bytebuffer are respected by the queue implementation (I think you already planned it this way), so that partial buffer contents are possible when based on those values - don't think this will be a problem in a read-once-by-implementation queue where the position of the current bytebuffer is modified only on removeXy(...) while forgetting all the previous contents. Even slice(int length) is perfect for this because there is no starting offset that may lay somewhere before the current bytebuffer's position. This also does not change anything for automatic disposal, because bytebuffers are just garbage collected, when there is no more reference to them (e.g. when all queues are read completelty and the references were subsequently removed in this process). > Another possible problem in introducing a new buffer type is overhead. > ByteBufferQueue is already composite, and its elements can be composite - > index calculation cost might neutralize the advantage of zero-copy. On the > other hand, we can keep the index calculation depth to two level at maximum > and to one level in most cases, so it might not be a big problem. :) Think so, the index calculation could be limited to ByteBuffer (of course adding some overhead when using duplicates) and a simple current_position inside ByteBufferQueue...right? > So, I'm up for introducing a new type. However, there was also a request > that we have to use ByteBuffer as first citizen. Please let me know again > if there's an issue with introducing a new type that we are missing. The duplicate-approach would respect this, as mentioned I don't see a need for a new type because ByteBuffer already comes with everything needed ootb because of it's random access and all-content-between-position-and-limit nature. > Yes, we need to reuse the utility class on both ByteBuffers and > ByteBufferQueues. However, if we introduce a new buffer type, we can > provide all of them to both ByteBufferQueue and the new buffer type. To > avoid code duplication, we will still have to keep static methods somewhere > and make them package private. Ok - however I have no idea what other buffer types would be required even in the long term. Of course, modular design is not a bad thing, though. > I am sorry about the users.. but consistency in naming is more important > IMHO. We can add getters and putters which delegates all operations to > removeXXX and offerXXX, but I guess it will make ByteBufferQueue look huge. > Do you have better idea? If so, that would be fantastic. > Sorry I don't, I was just wondering ;). For me it's ok to stick with the queue specific methods. The rest will be done by toByteBuffer() but noone should be encouraged to use this. > Very true. Both random access and first offset access should be provided > for all getters and putters, and then we have no problem with such a > protocol, right? (just to make sure I understood the problem correctly. :) > Absolutely right :) > My idea is to provide something similar to ANTLR. The difference is it > works in a binary level (or course including text level) and it generates > non-blocking MINA ProtocolDecoder. Once properly done, we could replace all > existing codec with this and the maintenance cost will drop down > dramatically. Instead we will have to maintain the decoder generator big > time, but it's more entertaining. :) > I thought about such a concept a little bit today and I am not yet convinced that this will be practical. I think the generator would be very complex because very different rule sets and transformations need to be implemented depending on the task. In some cases even callbacks will be inevitable. Also error hooks that generate some output and close the connection (or keep it, or jump to some other state etc.) will be challenging. Ok, this may be some fun for a while but it may also be just one step too much for a networking framework, that already is so easy to work with. In most cases I think it will be much easier to implement a protocol using switch() - also the proposed queues will make this a pleasure ;). Ok, maybe a generator can create more efficient code in some cases but it may not in every case (most protocols are different/exotic by nature), and as I said I'm not convinced that using the generator will be easier. However, if you decide to stick with it, I'll of course try it out someday ;). regards Daniel
