Re: CumulativeProtocolDecoder and UDP

[tunca]

https://issues.apache.org/jira/browse/DIRMINA-1033
is submitted.




--
View this message in context: 
http://apache-mina.10907.n7.nabble.com/CumulativeProtocolDecoder-and-UDP-tp18927p50292.html
Sent from the Apache MINA Developer Forum mailing list archive at Nabble.com.

Re: CumulativeProtocolDecoder and UDP

[Emmanuel Lécharny]

Le 03/03/16 09:09, tunca a écrit :
> Hi Emmanuel,
>
> I downloaded apache mina 2.0.13 and updated  NioDatagramSession METADATA
> creation code.
> Set the 4th parameter to true, then for UDP CumulativeProtocolDecoder
> started to work as expected.

Indeed.

Although doing so you create a dependency on a patched version of MINA.
That means if someone in the future switch to a newer version of MINA,
your code will simply break.

Here is what I suggest : fill a JIRA asking for exposing the MetaData
parameter, allowing teh developper (you, here) to set the flag to
something else than the defautl value.

Re: CumulativeProtocolDecoder and UDP

[tunca]

Hi Emmanuel,

I downloaded apache mina 2.0.13 and updated  NioDatagramSession METADATA
creation code.
Set the 4th parameter to true, then for UDP CumulativeProtocolDecoder
started to work as expected.

Thanks A lot.

class NioDatagramSession extends NioSession {
static final TransportMetadata METADATA = new
DefaultTransportMetadata("nio", "datagram", true, true,
InetSocketAddress.class, DatagramSessionConfig.class,
IoBuffer.class);




--
View this message in context: 
http://apache-mina.10907.n7.nabble.com/CumulativeProtocolDecoder-and-UDP-tp18927p50289.html
Sent from the Apache MINA Developer Forum mailing list archive at Nabble.com.

Re: CumulativeProtocolDecoder and UDP

[Emmanuel Lécharny]

Le 03/03/16 08:29, tunca a écrit :
> Hi Emmanuel,
>
> Thanks a lot, you find out the root cause of the problem I believe.
>
> I created my own version of CumulativeProtocolDecoder
> (http://pastebin.com/m0qmbEVS) and as you suggested removed the has
> fragmentation check.
> It still doesn't accumulate, however the second package arrives without
> accumulating for the same session.

Remove completely the initial check, includinhg the call to doDecode(),
otherwise it will be called twice, consuming the buffer.

Re: CumulativeProtocolDecoder and UDP

[tunca]

Hi Emmanuel,

Thanks a lot, you find out the root cause of the problem I believe.

I created my own version of CumulativeProtocolDecoder
(http://pastebin.com/m0qmbEVS) and as you suggested removed the has
fragmentation check.
It still doesn't accumulate, however the second package arrives without
accumulating for the same session.

Thanks
Tunca



--
View this message in context: 
http://apache-mina.10907.n7.nabble.com/CumulativeProtocolDecoder-and-UDP-tp18927p50287.html
Sent from the Apache MINA Developer Forum mailing list archive at Nabble.com.

Re: CumulativeProtocolDecoder and UDP

[Emmanuel Lécharny]

Le 02/03/16 07:38, tunca a écrit :
> Hi Emmanuel.
> http://pastebin.com/embed_iframe/EpL4WE4b
> This is the link to my doDecode method.
>
> A complete message starts with x01 and ends with x03.
> For test I send 2 udp messages.
> First Udp message starts with x01 
> Second Udp messages ends with x03.
>
> When I send first message doDecode message is called with the content of the
> first Udp message.
> the doDecode method returns false as the message is not complete.
>
> Then the second Udp message is sent.
> However doDecode method is still called with only First Udp message's
> content from in IoBuffer parameter.

Ok, I think I see what's going on. The
CumulativeProtocolDecoder.decode() method starts with :

public void decode(IoSession session, IoBuffer in,
ProtocolDecoderOutput out) throws Exception {
if (!session.getTransportMetadata().hasFragmentation()) {
while (in.hasRemaining()) {
if (!doDecode(session, in, out)) {
break;
}
}

return;
}

and for a UDP session, we initialize the transport metadata this way :

static final TransportMetadata METADATA = new
DefaultTransportMetadata("nio", "datagram", true, false,
InetSocketAddress.class, DatagramSessionConfig.class,
IoBuffer.class);


where the forth parameter is 'boolean fragmentation' :

public DefaultTransportMetadata(String providerName, String name,
boolean connectionless, boolean fragmentation,
Class addressType, Class sessionConfigType,
Class... envelopeTypes) {

It's set to 'false', so the decode method simply don't accumulate the
data in one single buffer.

There is no way to change the flag once the metadata instance has been
created.

What you can do is to define your own version of the
CumulativeProtocolDecoder, copying the original code and simply discard
the test on the transportMetadata.

   

Re: CumulativeProtocolDecoder and UDP

[tunca]

Hi Emmanuel.
http://pastebin.com/embed_iframe/EpL4WE4b
This is the link to my doDecode method.

A complete message starts with x01 and ends with x03.
For test I send 2 udp messages.
First Udp message starts with x01 
Second Udp messages ends with x03.

When I send first message doDecode message is called with the content of the
first Udp message.
the doDecode method returns false as the message is not complete.

Then the second Udp message is sent.
However doDecode method is still called with only First Udp message's
content from in IoBuffer parameter.
Tunca




--
View this message in context: 
http://apache-mina.10907.n7.nabble.com/CumulativeProtocolDecoder-and-UDP-tp18927p50281.html
Sent from the Apache MINA Developer Forum mailing list archive at Nabble.com.

Re: CumulativeProtocolDecoder and UDP

[Ashish]

Agree, that's why the protocol shift to Application layer, meaning you
emulate what TCP does like how many packets and their sequence to
order them at receiving layer for meaningful interpretation.

On Tue, Mar 1, 2016 at 11:01 AM, Mondain  wrote:
> Ashish, just be aware that with UDP, some of your data may never arrive;
> that's the nature of the protocol.
>
> Paul
>
> On Tue, Mar 1, 2016 at 1:48 PM Ashish  wrote:
>
>> TCP handles fragmentation at its level, but for UDP you have to do it
>> at application layer meaning UDP data has to carry message sequences
>> and then you merge them at receiving end. Here you packets can come in
>> different order do you got to keep them somewhere before complete
>> message is constructed.
>>
>>
>>
>> On Tue, Mar 1, 2016 at 7:58 AM, tunca  wrote:
>> > Our customer has strange requirement about merging multiple udp/TCP
>> packages
>> > to create a single  message.
>> > There is a well defined protocol that defines message boundaries.
>> > The decoder is working good with TCP packages.  It can create a single
>> > message from multiple TCP packages.
>> > However when a message is fragmented into multiple packages the do
>> doDecode
>> > method always gives the same ioBuffer.
>> > I'll try 2.0.13 next day.
>> > Thanks
>> >
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> http://apache-mina.10907.n7.nabble.com/CumulativeProtocolDecoder-and-UDP-tp18927p50274.html
>> > Sent from the Apache MINA Developer Forum mailing list archive at
>> Nabble.com.
>>
>>
>>
>> --
>> thanks
>> ashish
>>
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Re: CumulativeProtocolDecoder and UDP

[Mondain]

Ashish, just be aware that with UDP, some of your data may never arrive;
that's the nature of the protocol.

Paul

On Tue, Mar 1, 2016 at 1:48 PM Ashish  wrote:

> TCP handles fragmentation at its level, but for UDP you have to do it
> at application layer meaning UDP data has to carry message sequences
> and then you merge them at receiving end. Here you packets can come in
> different order do you got to keep them somewhere before complete
> message is constructed.
>
>
>
> On Tue, Mar 1, 2016 at 7:58 AM, tunca  wrote:
> > Our customer has strange requirement about merging multiple udp/TCP
> packages
> > to create a single  message.
> > There is a well defined protocol that defines message boundaries.
> > The decoder is working good with TCP packages.  It can create a single
> > message from multiple TCP packages.
> > However when a message is fragmented into multiple packages the do
> doDecode
> > method always gives the same ioBuffer.
> > I'll try 2.0.13 next day.
> > Thanks
> >
> >
> >
> >
> > --
> > View this message in context:
> http://apache-mina.10907.n7.nabble.com/CumulativeProtocolDecoder-and-UDP-tp18927p50274.html
> > Sent from the Apache MINA Developer Forum mailing list archive at
> Nabble.com.
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>

Re: CumulativeProtocolDecoder and UDP

[Ashish]

TCP handles fragmentation at its level, but for UDP you have to do it
at application layer meaning UDP data has to carry message sequences
and then you merge them at receiving end. Here you packets can come in
different order do you got to keep them somewhere before complete
message is constructed.



On Tue, Mar 1, 2016 at 7:58 AM, tunca  wrote:
> Our customer has strange requirement about merging multiple udp/TCP packages
> to create a single  message.
> There is a well defined protocol that defines message boundaries.
> The decoder is working good with TCP packages.  It can create a single
> message from multiple TCP packages.
> However when a message is fragmented into multiple packages the do doDecode
> method always gives the same ioBuffer.
> I'll try 2.0.13 next day.
> Thanks
>
>
>
>
> --
> View this message in context: 
> http://apache-mina.10907.n7.nabble.com/CumulativeProtocolDecoder-and-UDP-tp18927p50274.html
> Sent from the Apache MINA Developer Forum mailing list archive at Nabble.com.



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Re: CumulativeProtocolDecoder and UDP

[Emmanuel Lécharny]

Le 01/03/16 16:58, tunca a écrit :
> Our customer has strange requirement about merging multiple udp/TCP packages
> to create a single  message. 
> There is a well defined protocol that defines message boundaries. 
> The decoder is working good with TCP packages.  It can create a single
> message from multiple TCP packages. 
> However when a message is fragmented into multiple packages the do doDecode
> method always gives the same ioBuffer. 

Define fragmented in a UDP context : how do you manage the packet ordering ?
> I'll try 2.0.13 next day. 

It won't solve your issue. You most certainly handling the incoming
packets incorrectly somewhere in your IoHandler. Typically, you are not
in the same session when 2 UDP packets are coming, thus the IoBuffer,
which is stored into the session, is different for each incoming packet.

Re: CumulativeProtocolDecoder and UDP

[Emmanuel Lécharny]

Le 01/03/16 15:05, tunca a écrit :
> Hi Trustin,
> We're using Mina Core 2.0.9.

Have you checked with 2.0.13 ?

> The Udp Cumulative Protocol doesn't seem to accumulate.
> The Decoder is working with TCP however, with UDP the 
> The size of the IoBuffer doesn't increase.

Using such a protocol decoder does not make any sense in UDP : either
you receive the full message, or you receive nothing. Moreover, there is
no guarantee that a message N will arrive *before* a message N+1 in UDP.

What exactly are you trying to do ?

Re: CumulativeProtocolDecoder and UDP

[tunca]

Hi Trustin,
We're using Mina Core 2.0.9.
The Udp Cumulative Protocol doesn't seem to accumulate.
The Decoder is working with TCP however, with UDP the 
The size of the IoBuffer doesn't increase.
Best Regards
Tunca



--
View this message in context: 
http://apache-mina.10907.n7.nabble.com/CumulativeProtocolDecoder-and-UDP-tp18927p50272.html
Sent from the Apache MINA Developer Forum mailing list archive at Nabble.com.

Re: CumulativeProtocolDecoder and UDP

[Trustin Lee]

On Dec 14, 2007 2:41 AM, Trustin Lee [EMAIL PROTECTED] wrote:
 On Dec 13, 2007 11:35 PM, Frederic Soulier [EMAIL PROTECTED] wrote:
  Hi Trustin
 
  Thx for your answer. It makes sense.
 
  In our case UDP messages from clients are guaranteed to fit in 1 UDP
  packet therefore the fact that UDP packet #2 may arrive before UDP
  packet #1 is not relevant. At our level we just get message #2 before
  message #1 and they are completely independant.
  Obviously should the need arise to have larger messages from the
  client we'll have to reconstruct the messages using UDP packets in
  the right order.
 
  The 2nd interesting finding which is more disconcerting is that using
  MINA 1.1.3 there's effectively no accumulation by the
  CumulativeProtocolDecoder for UDP traffic.
  We skipped MINA 1.1.4 because of an issue with ProtocolCodecExection
  I reported.
  With MINA 1.1.5 the ProtocolCodecException works as per 1.1.3 but the
  CumulativeProtocolDecoder now accumulates data for UDP!
 
  Our app using 1.1.3 -- no accumulation for UDP
  Our app using 1.1.4 -- accumulation for UDP
 
  Could you confirm this change of behaviour?

 That's very weird.  If so, it's a compatibility bug.  At least, in
 2.0, it should do nothing if TransportMetadata.hasFragmentation()
 returns false.  I think I didn't touch a line in
 CumulativeProtocolDecoder in 1.x anyway.  Let me check it out and get
 back to you.

Yeah, it seems like nobody made change in CumulativeProtocolDecoder.
It's probably a side effect of some changed in other parts of the
core.  I don't know where it is though.  Let me fix this in 2.0 rather
than addressing it in 1.x, because it basically doesn't make sense to
a UDP application to send a partial message and we don't have any
means to figure out the transport has fragmentation or not.

Thanks again for your understanding.
Trustin
-- 
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6

Re: CumulativeProtocolDecoder and UDP

[Frederic Soulier]

Hi Trustin

Thx for your answer. It makes sense.

In our case UDP messages from clients are guaranteed to fit in 1 UDP  
packet therefore the fact that UDP packet #2 may arrive before UDP  
packet #1 is not relevant. At our level we just get message #2 before  
message #1 and they are completely independant.
Obviously should the need arise to have larger messages from the  
client we'll have to reconstruct the messages using UDP packets in  
the right order.


The 2nd interesting finding which is more disconcerting is that using  
MINA 1.1.3 there's effectively no accumulation by the  
CumulativeProtocolDecoder for UDP traffic.
We skipped MINA 1.1.4 because of an issue with ProtocolCodecExection  
I reported.
With MINA 1.1.5 the ProtocolCodecException works as per 1.1.3 but the  
CumulativeProtocolDecoder now accumulates data for UDP!


Our app using 1.1.3 -- no accumulation for UDP
Our app using 1.1.4 -- accumulation for UDP

Could you confirm this change of behaviour?


On 13 Dec 2007, at 04:07, Trustin Lee wrote:


It's no use to use CumulativeProtocolDecoder in UDP because UDP never
guarantees the order of packets.  Messages can be mixed up.  :)

HTH,
Trustin

On Dec 12, 2007 4:27 AM, Frederic Soulier  
[EMAIL PROTECTED] wrote:

Hi

I have an issue with Datagrams and the CumulativeProtocolDecoder
which doesn't seem to cumulate!
(working just fine with TCP).

I have a Decoder extending CumulativeProtocolDecoder.

protected boolean doDecode(IoSession session, ByteBuffer in,
ProtocolDecoderOutput out)
throws Exception
 {
 int startIndex = in.position(); // Save buffer position
 boolean done = doDecodeWithoutBackingUp(session, in, out);
 if (!done)
 {
 // Reset buffer to start as-if doDecodeWithoutBackingUp
had never run
 in.position(startIndex);
 }
 return done;
 }


protected boolean doDecodeWithoutBackingUp(IoSession ioSession,
ByteBuffer in, ProtocolDecoderOutput out)
 throws Exception
{
 if (in.remaining()  TYPE_SIZE)
 {
 return false; // Need more data, we don't have the type
of the message
 }
 else
 {
 type = in.get();
 if (type == TYPE_MSG)
 {
 if (in.remaining()  TRANSACTION_ID_SIZE)
 {
 return false; // Need more data, we don't have
the transaction id
 }
 else
 {
 transactionId = in.getInt();

 // do we have the message length?
 if (in.remaining() 
CodecUtils.MESSAGE_LENGTH_SIZE)
 {
return false; // Need more data, we  
don't have

the message length
 }
 else
 {
int dataLength = in.getInt(); // Read  
4-byte

int data length

 // do we have all the data?
 if (in.remaining()  dataLength)
 {
 // Here we're already read 9
bytes (1 byte + 1 int + 1 int) but
   ==   // we don't have all we need so
we return false and next time
 // around we'll have more
 return false; // Need more data,
we don't have the complete data
 }
 else
 {
 ** HERE WE PROCESS A FULL
MESSAGE **

return true; // Please carry  
on giving

us more data to decode
 }
 }
}
}
 else
{
  throw new MySuperDuperException();
 }
}
}

The buffer (in.remaining = 30)
I read 9 bytes from the buffer to be able to decide whether to read
more or not.
in.remaining = 20 but my data length is 50 so I need more bytes and
return false.
Another datagram is received in the meantime, I expect the new
datagram to be accumulated behind what I haven't processed
(considering I returned true).

Instead I only get the new datagram and the previous stuff I haven't
processed yet is gone to the big bit bucket in the sky!

Is the Cumulative stuff supposed to work with UDP the same way it
works with TCP? As I wrote at the top it works prefectly with TCP.

Thx.

--
Frederic P. Soulier
OpenPGP key available on http://pgpkeys.mit.edu/
1024D/BA6700ED   49A6 8E8E 4230 8D41 1ADE  B649 3203 1DD2 BA67 00ED






--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6


--
Frederic P. Soulier
OpenPGP key available on http://pgpkeys.mit.edu/
1024D/BA6700ED   49A6 8E8E 4230 8D41 1ADE  B649 3203 1DD2 BA67 00ED

Re: CumulativeProtocolDecoder and UDP

[Trustin Lee]

On Dec 13, 2007 11:35 PM, Frederic Soulier [EMAIL PROTECTED] wrote:
 Hi Trustin

 Thx for your answer. It makes sense.

 In our case UDP messages from clients are guaranteed to fit in 1 UDP
 packet therefore the fact that UDP packet #2 may arrive before UDP
 packet #1 is not relevant. At our level we just get message #2 before
 message #1 and they are completely independant.
 Obviously should the need arise to have larger messages from the
 client we'll have to reconstruct the messages using UDP packets in
 the right order.

 The 2nd interesting finding which is more disconcerting is that using
 MINA 1.1.3 there's effectively no accumulation by the
 CumulativeProtocolDecoder for UDP traffic.
 We skipped MINA 1.1.4 because of an issue with ProtocolCodecExection
 I reported.
 With MINA 1.1.5 the ProtocolCodecException works as per 1.1.3 but the
 CumulativeProtocolDecoder now accumulates data for UDP!

 Our app using 1.1.3 -- no accumulation for UDP
 Our app using 1.1.4 -- accumulation for UDP

 Could you confirm this change of behaviour?

That's very weird.  If so, it's a compatibility bug.  At least, in
2.0, it should do nothing if TransportMetadata.hasFragmentation()
returns false.  I think I didn't touch a line in
CumulativeProtocolDecoder in 1.x anyway.  Let me check it out and get
back to you.

Thanks for the report.
Trustin
-- 
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6

Re: CumulativeProtocolDecoder and UDP

[Trustin Lee]

It's no use to use CumulativeProtocolDecoder in UDP because UDP never
guarantees the order of packets.  Messages can be mixed up.  :)

HTH,
Trustin

On Dec 12, 2007 4:27 AM, Frederic Soulier [EMAIL PROTECTED] wrote:
 Hi

 I have an issue with Datagrams and the CumulativeProtocolDecoder
 which doesn't seem to cumulate!
 (working just fine with TCP).

 I have a Decoder extending CumulativeProtocolDecoder.

 protected boolean doDecode(IoSession session, ByteBuffer in,
 ProtocolDecoderOutput out)
 throws Exception
  {
  int startIndex = in.position(); // Save buffer position
  boolean done = doDecodeWithoutBackingUp(session, in, out);
  if (!done)
  {
  // Reset buffer to start as-if doDecodeWithoutBackingUp
 had never run
  in.position(startIndex);
  }
  return done;
  }


 protected boolean doDecodeWithoutBackingUp(IoSession ioSession,
 ByteBuffer in, ProtocolDecoderOutput out)
  throws Exception
 {
  if (in.remaining()  TYPE_SIZE)
  {
  return false; // Need more data, we don't have the type
 of the message
  }
  else
  {
  type = in.get();
  if (type == TYPE_MSG)
  {
  if (in.remaining()  TRANSACTION_ID_SIZE)
  {
  return false; // Need more data, we don't have
 the transaction id
  }
  else
  {
  transactionId = in.getInt();

  // do we have the message length?
  if (in.remaining() 
 CodecUtils.MESSAGE_LENGTH_SIZE)
  {
 return false; // Need more data, we don't have
 the message length
  }
  else
  {
 int dataLength = in.getInt(); // Read 4-byte
 int data length

  // do we have all the data?
  if (in.remaining()  dataLength)
  {
  // Here we're already read 9
 bytes (1 byte + 1 int + 1 int) but
==   // we don't have all we need so
 we return false and next time
  // around we'll have more
  return false; // Need more data,
 we don't have the complete data
  }
  else
  {
  ** HERE WE PROCESS A FULL
 MESSAGE **

 return true; // Please carry on giving
 us more data to decode
  }
  }
 }
 }
  else
 {
   throw new MySuperDuperException();
  }
 }
 }

 The buffer (in.remaining = 30)
 I read 9 bytes from the buffer to be able to decide whether to read
 more or not.
 in.remaining = 20 but my data length is 50 so I need more bytes and
 return false.
 Another datagram is received in the meantime, I expect the new
 datagram to be accumulated behind what I haven't processed
 (considering I returned true).

 Instead I only get the new datagram and the previous stuff I haven't
 processed yet is gone to the big bit bucket in the sky!

 Is the Cumulative stuff supposed to work with UDP the same way it
 works with TCP? As I wrote at the top it works prefectly with TCP.

 Thx.

 --
 Frederic P. Soulier
 OpenPGP key available on http://pgpkeys.mit.edu/
 1024D/BA6700ED   49A6 8E8E 4230 8D41 1ADE  B649 3203 1DD2 BA67 00ED





-- 
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6