On Thu, 18 Feb 2021 at 17:00, akabhishek1 <[email protected]> wrote: > > Hi Robbie, > > Thanks a lot for quick reply. We received only one below exception. There > are 7 destinations connected with single serviceBus connection. If links are > detached on “unknown peer error”, then we should receive at least 7 > exceptions at "onException" block, but received only one. > > I checked the log and find out that Application has not requested to close > JMS connection and does not initiate to close session/consumer of 6 > destinations, but initiate to close/create session/consumer of only one > destination. > > As Application received one exception, and found issue in only one > destination, So application re-established communication with only one > receiver (sub1). I am hoping closer of one destination (session/consumer) > shouldn't impact other session/consumer of 6 destinations. > > Big trouble points for us that we haven't received any exception for 6 > destinations and links were detached as well (tested with disabling the > queue which didn’t bubbled up error on onException block). > > Just FYI - All Seven destinations(sub1, sub2, sub3..) are subscriptions of > single topic ('TestTopic') > > Unfortunately I don't have any stacktrace, so i can't provide much details. > Please take a look below thread dumps, if it helps - First_App_ThreadDump > <https://github.com/abhikt48/servicebus/blob/master/For%20Qpid/Qpid-ServiceBus-threadDump.tdump> > > Going forward, we will log stacktrace in onException block.
Its hard to tell much from singular thread dumps, though there are some interesting things: The above file shows evidence of [at least remnants of] 18 connections, and lots and lots of sessions (say 150+) across them. This doesnt seem to match expectations from your overview, suggesting perhaps at least there is an issue with the cleanup and reestablishment bits. > > -------------------Details of another Application which have same issue like > above--------------- > > We are using latest jar 3 applications and we received this type of error in > two applications. I looked error on other application and that application > is very light, not heavier like above one. Please find below findings > > ##JMS Infrastructure## > JMS connection - 1 > Exception listener - 1 > 3 listener - (12 (3*4) consumers) > 7 publisher - > every consumer/publisher have separate JMS session, MessageConsumer and > producer > > Listener Details - > 1. QueueA > 2. QueueA/$DeadLetterQueue > 3. QueueB > > ####Actions after exception ### > 1. Exception on "onException" block - > Connection exception from 'SBus_Connector_1', isConnectionActive 'true', > Reason 'Unknown error from remote peer' - 'class javax.jms.JMSException', > cause 'org.apache.qpid.jms.provider.ProviderException: Unknown error from > remote peer' > > 2. Checked isConsumerActive for 'QueueA' - result “true” - (We have problem > here, not consuming messages) > 3. Checked isConsumerActive for 'QueueA/$DeadLetterQueue' - result “false”, > Re-established consumer > - I have doubt here, As 'QueueA/$DeadLetterQueue' consumer is > deactive, so > 'QueueA' consumer should be deactive as well. Please suggest your opinion. If you have 2 seperate consumers, if one gets closed it doesnt necessarily mean they are both. There is also nothing that means even if they both are, that it happens in a particular order at a particular time. > 4. Checked isConsumerActive for 'QueueB' - result “true” - (Did nothing) > > FYI - We re-establish consumer only if consumer is deactive. > > Same Application have another JMS connection(Separate ServiceBus broker > endpoint), on which we got same exception, > Error - Connection exception from 'SBus_Connector_2', isConnectionActive > 'true', Reason 'Unknown error from remote peer' - 'class > javax.jms.JMSException', cause > 'org.apache.qpid.jms.provider.ProviderException: Unknown error from remote > peer' > > This JMS connection has one publisher which is trying to publish message, > while publishing it also throws "ProviderException - Unknown error from > remote peer". > > Please find exception StackTrace - StackTrace_Unknown_Remote_Error.json > <https://github.com/abhikt48/servicebus/blob/master/For%20Qpid/StackTrace_Unknown_Remote_Error.json> > This is almost unreadable, and seems to omit line numbers for the stack, removing a good amount of useful info. > > ThreadDump - Application_Thread_Dump > <https://github.com/abhikt48/servicebus/blob/master/For%20Qpid/qpid-AppB-threadDump.tdump> > > ##Overall Obesrvation ## > 1. Exception listener does not receiver multiple notification for receiver > detachment. At least one notification should trigger for one destination. If there is actually a reason to, i.e they actually were closed, and e.g. if none of the reasons they wouldnt be occured already, e.g parent session/connection bits have already been closed for example. Cant really tell what happened from the limited detail available. > 2. JmsMessageConsumer.getMessageListener()->checkClosed() --> does not > provide correct closed status. It could be sync up issue. > > Suggestion/Question - We can't enable TRACE in PROD environment because it > emits lot of logs. Can you please tell us any qpid class which logs TRACE > event only on failure like "Unknown error from remote peer". So we can log > enable Qpid log atleast for failure. > The session has some info logging of e.g producer/consumer closures. I dont think there is one place to get minimal logging of those exceptions, the exceptions themselves are typically that indication. There is an 'connection listener' (using the JmsConnectionListener interface) that we use within tests at times which is called at the end of the overall process, though note asynchronously from the exception listener being called, you could perhaps use it to add your own additional extra logging. > > Please suggest your opinion on above query, So we can proceed further. I > will keep on working parallelly to replicate this issue if possible and > update you if found any. > > Regards, > Abhishek Kumar > > > > > > > > > > > > > > > -- > Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
