Re: MaxFrameSize not honored and allowing 64MB messages to go through

2016-08-17 Thread Matthewm
To make this a little more clear, the few items I am struggling to still
understand are these. Really appreciate any comments or ideas anyone has. 

>From the byte stream that’s incoming to the broker, the size provided in
bytes 21 to 24 before the Datatype parameter and the ActiveMQ magic, this is
the value that’s pulled for use to make sure its not beyond the configured
maxFrameSize. Are we missing a required configuration that’s allowing this
size to be a negative value or is that expected?  

byte|[21] |0
byte|[22] |0
byte|[23] |0
byte|[24] |-16

byte|[25] |1
byte|[26] |65 ->A
byte|[27] |99 ->c
byte|[28] |116 ->t
byte|[29] |105 ->i
byte|[30] |118 ->v
byte|[31] |101 ->e
byte|[32] |77 ->M
byte|[33] |81 ->Q

Later on in the stream, if it was parsed correctly, after the ActiveMQ magic
and the Version (10 in our example), the size used for the Byte array would
be parsed as 16mb. Far better than the 64mb its currently trying to allocate
for us, but still that size seems large for the data passed in. Again is
this a setting we are missing when the connection is created?   

ActiveMQ magic: 
byte|[26] |65 ->A
byte|[27] |99 ->c
byte|[28] |116 ->t
byte|[29] |105 ->i
byte|[30] |118 ->v
byte|[31] |101 ->e
byte|[32] |77 ->M
byte|[33] |81 ->Q
Version (10)
byte|[34] |0
byte|[35] |0
byte|[36] |0
byte|[37] |10
Byte array size (16777216)
byte|[38] |1
byte|[39] |0
byte|[40] |0
byte|[41] |0

We plan to test both with tightEncodingEnabled set to false, and also with
the 5.14 version, but I wanted to clear up my questions around this.  

We use the ActiveMQSslConnectionFactory and these are the parameters we
currently pass into this when the factory is created:

initialReconnectDelay=10
maxReconnectDelay=3
maxReconnectAttempts=10
startupMaxReconnectAttempts=1
timeout=6
randomize=false
useExponentialBackOff=true
priorityBackup=true
jms.useAsyncSend=true
nested.soTimeout=6
nested.soWriteTimeout=6
nested.wireFormat.maxFrameSize=262144





--
View this message in context: 
http://activemq.2283324.n4.nabble.com/MaxFrameSize-not-honored-and-allowing-64MB-messages-to-go-through-tp4715094p4715595.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.


Re: MaxFrameSize not honored and allowing 64MB messages to go through

2016-08-05 Thread jsuen
sorry, I forgot to mention the version...

We are using 5.11.1.  I diff between v10 and v11 and also 5.11.1 and 5.13.4
on the affected OpenWire code and they looked somewhat the same.

Thanks!



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/MaxFrameSize-not-honored-and-allowing-64MB-messages-to-go-through-tp4715094p4715096.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.


Re: MaxFrameSize not honored and allowing 64MB messages to go through

2016-08-05 Thread Christopher Shannon
What version are you using?  There have been various bug fixes and
improvements with this in later releases, especially 5.12.0 which
introduced OpenWire v11.  Since you are using OpenWire v10 I'm assuming you
have an older release so you should try upgrading.

On Fri, Aug 5, 2016 at 11:56 AM, jsuen  wrote:

> I am trying to drop any large messages by setting maxFrameSize=262144 and
> it
> seems that it negotiated the setting.
>
> 2016-07-25T12:52:10.040-0700  CEF:1 | good |
> wrap_mvn_com.good.activemq_osgi.activemq-gems_5.11.1 | 0.0.0 | DEBUG |
> unknown | 3 | ID=147 THR=.137:61616@39196 CAT=WireFormatNegotiator
> MSG=tcp://JSUEN-W520.rim.net/10.100.153.137:61616@39196 after negotiation:
> OpenWireFormat{version=10, cacheEnabled=true, stackTraceEnabled=true,
> tightEncodingEnabled=true, sizePrefixDisabled=false, maxFrameSize=262144}
>
> However, I am still seeing 64MB messages go through...
>
> at
> org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.
> tightUnmarshalByteSequence
> at org.apache.activemq.openwire.v10.WireFormatInfoMarshaller.
> tightUnmarshal
> at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal
> at org.apache.activemq.openwire.OpenWireFormat.unmarshal
>
> The activemq code that does decide the size of the byte[] does use the
> readInt() call to simply do this:
> 386 public final int readInt() throws IOException {
> 387 int ch1 = in.read();
> 388 int ch2 = in.read();
> 389 int ch3 = in.read();
> 390 int ch4 = in.read();
> 391 if ((ch1 | ch2 | ch3 | ch4) < 0)
> 392 throw new EOFException();
> 393 return ((ch1 << 24) + (ch2 << 16) + (ch3 << 8) + (ch4 << 0));
> Each instance in every dump the data in positions 104 to 107 end up giving
> us the invalid byte[] size in tightUnmarshalByteSequence.
> So going back to that internalBuffer byte array, if we grab the values from
> POS 104 to 107:
> Type|Name |Value
> 
> byte|[104]|4
> byte|[105]|0
> byte|[106]|0
> byte|[107]|17
> byte|[108]|83
> byte|[109]|116
> byte|[110]|97
> byte|[111]|99
> byte|[112]|107
> byte|[113]|84
> byte|[114]|114
> 
> And do the left bit shift operations we get:
> 4 << 24 = 67108864
> 0 << 16 = 0
> 0 << 16 = 0
> 17 << 0 = 17
> Gives us a sum of 67,108,881 that match’s the array size: byte[67108881].
>
> Debugger is showing the maxFrameSize on OpenWireFormat set to 262144, and
> this is one of the threads using 64mb mem. But if we dive down into its
> command.WireFormatInfo propreties hash, we see the max frame size is wrong.
>
> If we dump out all the properties of the OpenWireFormat for this one thread
> they read as:
>
> KEY |Value
> 
> CacheSize   |1024
> SizePrefixDisabled  |false
> TcpNoDelayEnabled   |true
> StackTraceEnabled   |true
> CacheEnabled|true
> MaxFrameSize|9223372036854775807
> TightEncodingEnabled|true
> MaxInactivityDuration   |3
> MaxInactivityDurationInitalDelay|1
> 
>
>
> Note, this occurs only when one of the node become the new broker.
> How can we make sure all the openwire settings are correctly negotiated
> between client and broker?
>
>
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/MaxFrameSize-not-honored-and-allowing-64MB-
> messages-to-go-through-tp4715094.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>