Hi JB, We took the jetty jar from 5.16.0 (which does not have the issue) and replaced the jetty jar in 5.16.1. Tcpdump confirms that we were testing jetty 9.4.28. This does not resolve the issue. That said, we didn’t change any of the other related jetty files, so there could still be something in jetty.
We believe this has something to do with the Content-Type flag (but still unclear why the difference in ActiveMQ versions). By default, these are different in Postman and curl. The ultimate customer issue is in WebLogic, but I don’t have access to that for testing. Best Regards, Doug Whitfield From: Jean-Baptiste Onofre <j...@nanthrax.net> Date: Tuesday, June 15, 2021 at 10:38 PM To: dev@activemq.apache.org <dev@activemq.apache.org> Subject: Re: Truncated messages in 5.16.1 when using Postman Hi, I will try to reproduce, so, I will start from ActiveMQ "vanilla" distribution (I’m suspecting something related to Jetty update). I will use curl to submit message. Just to be sure, what transport connector are you using: http, https, ws, … to post the message (the full URL would be helpful) ? Thanks, Regards JB > Le 15 juin 2021 à 22:33, Doug Whitfield <dwhitfi...@perforce.com> a écrit : > > Hi JB, > > The test instance AP is using is completely stock (downloaded from the > ActiveMQ website). We will confirm with the customer their jetty.xml > situation (I’m skeptical, since they see the exact same jump in version that > we do), but there’s something strange going on in the default build. > > You mention that we would get a message in the log regarding maxFrameSize. We > see no such message. Would we need a non-standard debugging level to see such > a message? > > Is there anything that would be useful for us to send, like a tcpdump? > > Best Regards, > > Doug Whitfield > > > > From: Jean-Baptiste Onofre <j...@nanthrax.net <mailto:j...@nanthrax.net>> > Date: Tuesday, June 15, 2021 at 9:19 AM > To: dev@activemq.apache.org <mailto:dev@activemq.apache.org> > <dev@activemq.apache.org <mailto:dev@activemq.apache.org>> > Subject: Re: Truncated messages in 5.16.1 when using Postman > But you go throw the HTTP connector right ? > Not tcp. > > So, do you have maxFrameSize on the http connector ? And also, do you use a > jetty.xml dedicated for the http connector (it’s not the same as > conf/jetty.xml used by the admin web console). > > Regards > JB > >> Le 15 juin 2021 à 14:58, Andrew Pomponio <apompo...@perforce.com> a écrit : >> >> Yes, the message size that seems to start the issue is 65,536 b or 65k in >> size. Smaller than that and the issue does not occur, we tried with a 32k >> message and did not have issues. >> >> On 6/15/21, 6:37 AM, "Jean-Baptiste Onofre" <j...@nanthrax.net> wrote: >> >> OK, and the message you sent is smaller than 104857600 b ? >> >> Anyway, if the message is greater than the maxFrameSize, you have a >> message in the activemq log and an exception on the client side >> (JMSException). >> >> Regards >> JB >> >>> Le 15 juin 2021 à 13:53, Andrew Pomponio <apompo...@perforce.com> a écrit : >>> >>> Hello Jean-Baptiste, thank you so much for your reply, >>> >>> For testing, we've been using stock defaults, for example, in our test for >>> 5.15.14 we used maxFrameSize=104857600. >>> >>> On 6/15/21, 5:48 AM, "Jean-Baptiste Onofre" <j...@nanthrax.net> wrote: >>> >>> Hi, >>> >>> Sorry I missed the message. >>> >>> Maybe you have reach the maxFrameSize on the transport connector. Did you >>> try to increase it ? >>> >>> What’s your transport connector configuration in activemq.xml ? >>> >>> Regards >>> JB >>> >>>> Le 15 juin 2021 à 13:35, Andrew Pomponio <apompo...@perforce.com> a écrit : >>>> >>>> Dear ActiveMQ Community, >>>> >>>> An update on the issue, we've done continued testing and found that this >>>> issue starts as far back as 5.15.14 and is also present in 5.16.2. >>>> >>>> On 6/10/21, 8:57 AM, "Andrew Pomponio" <apompo...@perforce.com> wrote: >>>> >>>> Dear ActiveMQ Community, >>>> >>>> I am reaching out on behalf of a customer of ours experiencing what >>>> appears to be a bug (or new default limitation) in an upgraded version of >>>> AMQ from the version they are currently using. We have tested this >>>> behavior in our lab environment and are able to replicate what they are >>>> seeing using the following environmental components; >>>> >>>> ActiveMQ 5.15.11 (works) and 5.16.1 (does not work) >>>> Postman 8.6.1 >>>> 72k sized payload (the exact limitations appears to be 65536 bytes) >>>> >>>> The following test is to be run once on 5.15 and then on 5.16. >>>> >>>> First, create a payload of random gibberish by executing the following; >>>> >>>> dd bs=1024 count=71 </dev/random >payload-test72k.txt >>>> >>>> cat the file and copy the contents (gibberish) into the <ns2:Data> tags >>>> like so; >>>> >>>> <ns2:NormalizedMessage xmlns:ns2="http://private <http://private/>" >>>> xmlns:ns4="http://differentprivate >>>> <http://differentprivate/>"><ns2:BaseDescriptor><ns2:MsgID>AVSN#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID<http://differentprivate%22%3e%3cns2:BaseDescriptor%3e%3cns2:MsgID%3eAVSN#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID<http://differentprivate/%3e%22%3e%3cns2:BaseDescriptor%3e%3cns2:MsgID%3eAVSN#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID<http://differentprivate%22%3e%3cns2:BaseDescriptor%3e%3cns2:MsgID%3eAVSN#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID>> >>>> >>>> <http://differentprivate";><ns2:basedescriptor><ns2:msgid>avsn#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID<http://differentprivate%22%3e%3cns2:basedescriptor%3e%3cns2:msgid%3eavsn#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID>>>>over >>>> 9000</ns2:SourceApplicationID><ns2:SourceUserID>over >>>> 9000</ns2:SourceUserID><ns2:SequenceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SequenceTimeStamp> >>>> >>>> </ns2:BaseDescriptor><ns2:RoutingDescriptor><ns4:RoutingInfo><ns4:ApplicationName>over >>>> 9000</ns4:ApplicationName><ns4:ApplicationCode>over >>>> 9000</ns4:ApplicationCode><ns4:ApplicationVersion>1.0</ns4:ApplicationVersion><ns4:PublishTimeStamp>2021-04-28T15:28:57.294Z</ns4:PublishTimeStamp><ns4:Status>true</ns4:Status><ns4:StatusDescription>DocumentUploadRequest</ns4:StatusDescription> >>>> </ns4:RoutingInfo> >>>> </ns2:RoutingDescriptor><ns2:lolDescriptor><ns2:lolReferenceID>MEDDOCINT042803</ns2:lolReferenceID><ns2:AliaslolReferenceIDs><ns2:lolReferenceID>MEDDOCINT042803</ns2:lolReferenceID> >>>> >>>> </ns2:AliaslolReferenceIDs><ns2:lolFolderID>0</ns2:lolFolderID><ns2:ClientFileID></ns2:ClientFileID><ns2:nopeCompany><ns2:VantiveCode>QO</ns2:VantiveCode><ns2:nahbroCustID>0</ns2:nahbroCustID> >>>> </ns2:nopeCompany> </ns2:lolDescriptor><ns2:Payload><ns2:Data>[…] >>>> �M��pt=��/�s >>>> .7u�a��F�{�S���Zt��R�^&�(/k�*!y���\��}I�i >>>> �� >>>> �؍I�����ՙ�y1 >>>> ��2��˧v�i� >>>> ���-F�ʅ�"��5�ԍ;C�+V[…]</ns2:Data> </ns2:Payload> >>>> </ns2:NormalizedMessage> >>>> >>>> Copy the above XML into the raw body of Postman. The XML and payload you >>>> copy and pasted from payload-test72k.txt is going to >>>> http://localhost:8161/api/message/orders.input/. With ActiveMQ running in >>>> debug mode, using Postman, with basic authorization login, click POST. You >>>> should receive an output of message sent, though the message will not show >>>> up in the UI of AMQ. We do not need it to for the purposes of this >>>> experiment. >>>> >>>> Back in a terminal from the bin directory in AMQ; >>>> >>>> tail -200 ../data/activemq.log >>>> >>>> You then can see in the log entries two different examples of the message, >>>> one that shows the full body of text being sent to ActiveMQ, and another >>>> that shows a condensed summary of the message being sent. >>>> >>>> 2021-06-10 07:50:44,510 | DEBUG | Sent! to destination: >>>> topic://orders.input. message: ActiveMQTextMessage {commandId = 0, >>>> responseRequired = false, messageId = >>>> ID:apomponio-56497-1623331633553-4:1:1:1:6, originalDestination = null, >>>> originalTransactionId = null, producerId = null, destination = >>>> topic://orders.input., transactionId = null, expiration = 0, timestamp = >>>> 1623333044509, arrival = 0, brokerInTime = 0, brokerOutTime = 0, >>>> correlationId = null, replyTo = null, persistent = true, type = null, >>>> priority = 5, groupID = null, groupSequence = 0, targetConsumerId = null, >>>> compressed = false, userID = null, content = null, marshalledProperties = >>>> null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = >>>> null, readOnlyProperties = false, readOnlyBody = false, droppable = false, >>>> jmsXGroupFirstForConsumer = false, text = <ns2:NormalizedMessage >>>> xmlns:ns2="http://priv...�T�T�Z5� �] >>>> } | org.apache.activemq.web.WebClient | qtp632475595-35 >>>> >>>> In the example above from the 5.16 server, we see in the text = field the >>>> start of our message, and right after http://priv we see … followed by the >>>> portion of gibberish where the message ends in the log. In our 5.15 logs >>>> we see http://priv...zedMessage> showing us the partial ending XML tag of >>>> </ns2:NormalizedMessage> which is the expected behavior. >>>> >>>> In 5.15.11, the full length of the body message shows up in the >>>> activemq.log file, but in 5.16.1 that same message sent exactly the same >>>> way is truncated around the 64k size. >>>> >>>> Is this a known default limitation introduced in 5.16? If so, how do we >>>> increase it? >>>> >>>> If this is anticipated to be a bug, what additional information, if any, >>>> do you need from us? Also, in the future, would something like this be >>>> more appropriate for the users mailing list instead? >>>> >>>> >>>> >>>> This e-mail may contain information that is privileged or confidential. If >>>> you are not the intended recipient, please delete the e-mail and any >>>> attachments and notify us immediately. >>>> >>> >>> >>> >>> CAUTION: This email originated from outside of the organization. Do not >>> click on links or open attachments unless you recognize the sender and know >>> the content is safe. >>> >>> >>> This e-mail may contain information that is privileged or confidential. If >>> you are not the intended recipient, please delete the e-mail and any >>> attachments and notify us immediately. >>> >> >> >> >> CAUTION: This email originated from outside of the organization. Do not >> click on links or open attachments unless you recognize the sender and know >> the content is safe. >> >> >> This e-mail may contain information that is privileged or confidential. If >> you are not the intended recipient, please delete the e-mail and any >> attachments and notify us immediately. >> > > > > CAUTION: This email originated from outside of the organization. Do not click > on links or open attachments unless you recognize the sender and know the > content is safe. > > > This e-mail may contain information that is privileged or confidential. If > you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately. CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
