Hi Doug, Sorry, but I don’t fully understand what you are doing ;)
My understanding (correct me if I’m wrong) is: 1. You have a http related transport connector in activemq.xml (<transportConnector name="http" uri="http://0.0.0.0:8888 <http://0.0.0.0:8888/>"/>) 2. You push a message via a connection factory ? Or you go through the REST API ? 3. The message you consume is incomplete (don’t consider the log message, but the consumed message) Is that correct ? Regards JB > Le 16 juin 2021 à 17:31, Doug Whitfield <dwhitfi...@perforce.com> a écrit : > > Hi JB, > > We took the jetty jar from 5.16.0 (which does not have the issue) and > replaced the jetty jar in 5.16.1. Tcpdump confirms that we were testing jetty > 9.4.28. This does not resolve the issue. That said, we didn’t change any of > the other related jetty files, so there could still be something in jetty. > > We believe this has something to do with the Content-Type flag (but still > unclear why the difference in ActiveMQ versions). By default, these are > different in Postman and curl. The ultimate customer issue is in WebLogic, > but I don’t have access to that for testing. > > Best Regards, > Doug Whitfield > > From: Jean-Baptiste Onofre <j...@nanthrax.net <mailto:j...@nanthrax.net>> > Date: Tuesday, June 15, 2021 at 10:38 PM > To: dev@activemq.apache.org <mailto:dev@activemq.apache.org> > <dev@activemq.apache.org <mailto:dev@activemq.apache.org>> > Subject: Re: Truncated messages in 5.16.1 when using Postman > Hi, > > I will try to reproduce, so, I will start from ActiveMQ "vanilla" > distribution (I’m suspecting something related to Jetty update). > > I will use curl to submit message. Just to be sure, what transport connector > are you using: http, https, ws, … to post the message (the full URL would be > helpful) ? > > Thanks, > Regards > JB > >> Le 15 juin 2021 à 22:33, Doug Whitfield <dwhitfi...@perforce.com> a écrit : >> >> Hi JB, >> >> The test instance AP is using is completely stock (downloaded from the >> ActiveMQ website). We will confirm with the customer their jetty.xml >> situation (I’m skeptical, since they see the exact same jump in version that >> we do), but there’s something strange going on in the default build. >> >> You mention that we would get a message in the log regarding maxFrameSize. >> We see no such message. Would we need a non-standard debugging level to see >> such a message? >> >> Is there anything that would be useful for us to send, like a tcpdump? >> >> Best Regards, >> >> Doug Whitfield >> >> >> >> From: Jean-Baptiste Onofre <j...@nanthrax.net <mailto:j...@nanthrax.net> >> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> >> Date: Tuesday, June 15, 2021 at 9:19 AM >> To: dev@activemq.apache.org <mailto:dev@activemq.apache.org> >> <mailto:dev@activemq.apache.org <mailto:dev@activemq.apache.org>> >> <dev@activemq.apache.org <mailto:dev@activemq.apache.org> >> <mailto:dev@activemq.apache.org <mailto:dev@activemq.apache.org>>> >> Subject: Re: Truncated messages in 5.16.1 when using Postman >> But you go throw the HTTP connector right ? >> Not tcp. >> >> So, do you have maxFrameSize on the http connector ? And also, do you use a >> jetty.xml dedicated for the http connector (it’s not the same as >> conf/jetty.xml used by the admin web console). >> >> Regards >> JB >> >>> Le 15 juin 2021 à 14:58, Andrew Pomponio <apompo...@perforce.com >>> <mailto:apompo...@perforce.com>> a écrit : >>> >>> Yes, the message size that seems to start the issue is 65,536 b or 65k in >>> size. Smaller than that and the issue does not occur, we tried with a 32k >>> message and did not have issues. >>> >>> On 6/15/21, 6:37 AM, "Jean-Baptiste Onofre" <j...@nanthrax.net >>> <mailto:j...@nanthrax.net>> wrote: >>> >>> OK, and the message you sent is smaller than 104857600 b ? >>> >>> Anyway, if the message is greater than the maxFrameSize, you have a >>> message in the activemq log and an exception on the client side >>> (JMSException). >>> >>> Regards >>> JB >>> >>>> Le 15 juin 2021 à 13:53, Andrew Pomponio <apompo...@perforce.com >>>> <mailto:apompo...@perforce.com>> a écrit : >>>> >>>> Hello Jean-Baptiste, thank you so much for your reply, >>>> >>>> For testing, we've been using stock defaults, for example, in our test for >>>> 5.15.14 we used maxFrameSize=104857600. >>>> >>>> On 6/15/21, 5:48 AM, "Jean-Baptiste Onofre" <j...@nanthrax.net >>>> <mailto:j...@nanthrax.net>> wrote: >>>> >>>> Hi, >>>> >>>> Sorry I missed the message. >>>> >>>> Maybe you have reach the maxFrameSize on the transport connector. Did you >>>> try to increase it ? >>>> >>>> What’s your transport connector configuration in activemq.xml ? >>>> >>>> Regards >>>> JB >>>> >>>>> Le 15 juin 2021 à 13:35, Andrew Pomponio <apompo...@perforce.com >>>>> <mailto:apompo...@perforce.com>> a écrit : >>>>> >>>>> Dear ActiveMQ Community, >>>>> >>>>> An update on the issue, we've done continued testing and found that this >>>>> issue starts as far back as 5.15.14 and is also present in 5.16.2. >>>>> >>>>> On 6/10/21, 8:57 AM, "Andrew Pomponio" <apompo...@perforce.com >>>>> <mailto:apompo...@perforce.com>> wrote: >>>>> >>>>> Dear ActiveMQ Community, >>>>> >>>>> I am reaching out on behalf of a customer of ours experiencing what >>>>> appears to be a bug (or new default limitation) in an upgraded version of >>>>> AMQ from the version they are currently using. We have tested this >>>>> behavior in our lab environment and are able to replicate what they are >>>>> seeing using the following environmental components; >>>>> >>>>> ActiveMQ 5.15.11 (works) and 5.16.1 (does not work) >>>>> Postman 8.6.1 >>>>> 72k sized payload (the exact limitations appears to be 65536 bytes) >>>>> >>>>> The following test is to be run once on 5.15 and then on 5.16. >>>>> >>>>> First, create a payload of random gibberish by executing the following; >>>>> >>>>> dd bs=1024 count=71 </dev/random >payload-test72k.txt >>>>> >>>>> cat the file and copy the contents (gibberish) into the <ns2:Data> tags >>>>> like so; >>>>> >>>>> <ns2:NormalizedMessage xmlns:ns2="http://private <http://private/> >>>>> <http://private/ <http://private/>>" xmlns:ns4="http://differentprivate >>>>> <http://differentprivate/> <http://differentprivate/ >>>>> <http://differentprivate/>>"><ns2:BaseDescriptor><ns2:MsgID>AVSN#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID<http://differentprivate%22%3e%3cns2:BaseDescriptor%3e%3cns2:MsgID%3eAVSN#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID<http://differentprivate/%3e%22%3e%3cns2:BaseDescriptor%3e%3cns2:MsgID%3eAVSN#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID<http://differentprivate%22%3e%3cns2:BaseDescriptor%3e%3cns2:MsgID%3eAVSN#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID> >>>>> >>>>> <http://differentprivate";><ns2:basedescriptor><ns2:msgid>avsn#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID<http://differentprivate/%3e%22%3e%3cns2:BaseDescriptor%3e%3cns2:MsgID%3eAVSN#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID<http://differentprivate%22%3e%3cns2:BaseDescriptor%3e%3cns2:MsgID%3eAVSN#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID>>> >>>>> <http://differentprivate >>>>> <http://differentprivate/>"><ns2:basedescriptor><ns2:msgid>avsn#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID<http://differentprivate%22%3e%3cns2:basedescriptor%3e%3cns2:msgid%3eavsn#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID> >>>>> >>>>> <http://differentprivate%22%3e%3cns2:basedescriptor%3e%3cns2:msgid%3eavsn#I64534#ajz123403252016#2016-03-25T13:15:55</ns2:MsgID><ns2:MsgType>INTF</ns2:MsgType><ns2:MsgMode>INBOUND</ns2:MsgMode><ns2:SourceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SourceTimeStamp><ns2:SourceApplicationID>>>>>over >>>>> 9000</ns2:SourceApplicationID><ns2:SourceUserID>over >>>>> 9000</ns2:SourceUserID><ns2:SequenceTimeStamp>2021-04-28T15:28:57.294Z</ns2:SequenceTimeStamp> >>>>> >>>>> </ns2:BaseDescriptor><ns2:RoutingDescriptor><ns4:RoutingInfo><ns4:ApplicationName>over >>>>> 9000</ns4:ApplicationName><ns4:ApplicationCode>over >>>>> 9000</ns4:ApplicationCode><ns4:ApplicationVersion>1.0</ns4:ApplicationVersion><ns4:PublishTimeStamp>2021-04-28T15:28:57.294Z</ns4:PublishTimeStamp><ns4:Status>true</ns4:Status><ns4:StatusDescription>DocumentUploadRequest</ns4:StatusDescription> >>>>> </ns4:RoutingInfo> >>>>> </ns2:RoutingDescriptor><ns2:lolDescriptor><ns2:lolReferenceID>MEDDOCINT042803</ns2:lolReferenceID><ns2:AliaslolReferenceIDs><ns2:lolReferenceID>MEDDOCINT042803</ns2:lolReferenceID> >>>>> >>>>> </ns2:AliaslolReferenceIDs><ns2:lolFolderID>0</ns2:lolFolderID><ns2:ClientFileID></ns2:ClientFileID><ns2:nopeCompany><ns2:VantiveCode>QO</ns2:VantiveCode><ns2:nahbroCustID>0</ns2:nahbroCustID> >>>>> </ns2:nopeCompany> </ns2:lolDescriptor><ns2:Payload><ns2:Data>[…] >>>>> �M��pt=��/�s >>>>> .7u�a��F�{�S���Zt��R�^&�(/k�*!y���\��}I�i >>>>> �� >>>>> �؍I�����ՙ�y1 >>>>> ��2��˧v�i� >>>>> ���-F�ʅ�"��5�ԍ;C�+V[…]</ns2:Data> </ns2:Payload> >>>>> </ns2:NormalizedMessage> >>>>> >>>>> Copy the above XML into the raw body of Postman. The XML and payload you >>>>> copy and pasted from payload-test72k.txt is going to >>>>> http://localhost:8161/api/message/orders.input/. With ActiveMQ running in >>>>> debug mode, using Postman, with basic authorization login, click POST. >>>>> You should receive an output of message sent, though the message will not >>>>> show up in the UI of AMQ. We do not need it to for the purposes of this >>>>> experiment. >>>>> >>>>> Back in a terminal from the bin directory in AMQ; >>>>> >>>>> tail -200 ../data/activemq.log >>>>> >>>>> You then can see in the log entries two different examples of the >>>>> message, one that shows the full body of text being sent to ActiveMQ, and >>>>> another that shows a condensed summary of the message being sent. >>>>> >>>>> 2021-06-10 07:50:44,510 | DEBUG | Sent! to destination: >>>>> topic://orders.input. message: ActiveMQTextMessage {commandId = 0, >>>>> responseRequired = false, messageId = >>>>> ID:apomponio-56497-1623331633553-4:1:1:1:6, originalDestination = null, >>>>> originalTransactionId = null, producerId = null, destination = >>>>> topic://orders.input., transactionId = null, expiration = 0, timestamp = >>>>> 1623333044509, arrival = 0, brokerInTime = 0, brokerOutTime = 0, >>>>> correlationId = null, replyTo = null, persistent = true, type = null, >>>>> priority = 5, groupID = null, groupSequence = 0, targetConsumerId = null, >>>>> compressed = false, userID = null, content = null, marshalledProperties = >>>>> null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = >>>>> null, readOnlyProperties = false, readOnlyBody = false, droppable = >>>>> false, jmsXGroupFirstForConsumer = false, text = <ns2:NormalizedMessage >>>>> xmlns:ns2="http://priv...�T�T�Z5� �] >>>>> } | org.apache.activemq.web.WebClient | qtp632475595-35 >>>>> >>>>> In the example above from the 5.16 server, we see in the text = field the >>>>> start of our message, and right after http://priv we see … followed by >>>>> the portion of gibberish where the message ends in the log. In our 5.15 >>>>> logs we see http://priv...zedMessage> showing us the partial ending XML >>>>> tag of </ns2:NormalizedMessage> which is the expected behavior. >>>>> >>>>> In 5.15.11, the full length of the body message shows up in the >>>>> activemq.log file, but in 5.16.1 that same message sent exactly the same >>>>> way is truncated around the 64k size. >>>>> >>>>> Is this a known default limitation introduced in 5.16? If so, how do we >>>>> increase it? >>>>> >>>>> If this is anticipated to be a bug, what additional information, if any, >>>>> do you need from us? Also, in the future, would something like this be >>>>> more appropriate for the users mailing list instead? >>>>> >>>>> >>>>> >>>>> This e-mail may contain information that is privileged or confidential. >>>>> If you are not the intended recipient, please delete the e-mail and any >>>>> attachments and notify us immediately. >>>>> >>>> >>>> >>>> >>>> CAUTION: This email originated from outside of the organization. Do not >>>> click on links or open attachments unless you recognize the sender and >>>> know the content is safe. >>>> >>>> >>>> This e-mail may contain information that is privileged or confidential. If >>>> you are not the intended recipient, please delete the e-mail and any >>>> attachments and notify us immediately. >>>> >>> >>> >>> >>> CAUTION: This email originated from outside of the organization. Do not >>> click on links or open attachments unless you recognize the sender and know >>> the content is safe. >>> >>> >>> This e-mail may contain information that is privileged or confidential. If >>> you are not the intended recipient, please delete the e-mail and any >>> attachments and notify us immediately. >>> >> >> >> >> CAUTION: This email originated from outside of the organization. Do not >> click on links or open attachments unless you recognize the sender and know >> the content is safe. >> >> >> This e-mail may contain information that is privileged or confidential. If >> you are not the intended recipient, please delete the e-mail and any >> attachments and notify us immediately. > > > > CAUTION: This email originated from outside of the organization. Do not click > on links or open attachments unless you recognize the sender and know the > content is safe. > > > This e-mail may contain information that is privileged or confidential. If > you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately.
