Hello Matt ! I've executed a tcpdump in the karaf pod, and I observe very strange things (see attached png file). After the initial connection the client and server periodically use KeepAliveInfo messages. That seems to be a normal openWire dialog. But later, the client (10.153.50.183) changes and sends a RemoveInfo that seems to be accepted by the server. Then a ShutdownInfo is sent, and then the tcp closing sequence of fin/ack terminates the socket. Then a new socket is opened and a new openWire dialog begins.
Why does the client decide to break the keep alive dialog after 30 minutes ? Is there a default option that is applied in this configuration that we've never observed before with exactly the same configuration files ? Thanks again. Regards. Le lun. 10 juil. 2023 à 18:27, Matt Pavlovich <mattr...@gmail.com> a écrit : > > Hi Ephemeris- > > 1. Yes, you should use failover:() even with one URL (esp if using load > balancer) > > 2. These Camel errors are trick to troubleshoot without seeing all the > settings. Camel JMS uses Spring JMS under the covers, and understanding how > that works in combination with Connection Pooling is tricky. Specifically— > setting the max pooled objects is _not_ the same as max connections. Pooled > objects may include connections + sessions + consumers, depending on your > configuration. Setting that number to ’25’ may be too low. If your used > connections + sessions + consumers is 25 or higher. I think the default is > 500, so you are significantly lowering it. > > Adding more debug and add’l logging would be need to root cause. > > Also note, this level of debugging is beyond what is reasonable to expect > getting a resolution from ActiveMQ user mailing list and/or Stackoverflow > type developer support. > > Thanks, > Matt Pavlovich > > > On Jul 10, 2023, at 4:41 AM, Ephemeris Lappis <ephemeris.lap...@gmail.com> > > wrote: > > > > Hello again. > > > > Perhaps you've already seen my previous mail... I've tried some > > options and none of them seems to fix my connection issues. > > > > I've tried first adding "failover" over my tcp URL, and then > > "keekAlice=true" on the tcp transport, and then the two changes > > together, and I have exactly the same errors. > > > > My connection factory configuration now is : > > > > # Connection configuration > > type=activemq > > connectionFactoryType=ConnectionFactory > > > > # Names > > name=alice-jms > > osgi.jndi.service.name=jms/alice > > > > # Connection factory properties > > jms.url=failover:(tcp://alice-echanges-activemq:61616?keepAlive=true) > > jms.user=alice > > jms.password=alice > > jms.clientIDPrefix=CATERPILLAR > > > > # Set XA transaction > > xa=false > > > > # Connection pooling > > pool=pooledjms > > # Maximum number of connections for each user+password (default 1) > > pool.maxConnections=256 > > # Maximum idle time in seconds (default 30 seconds) > > pool.connectionIdleTimeout=30 > > # Interval for connections idle time checking in milliseconds (default > > 0 for none) > > pool.connectionCheckInterval=15000 > > > > Except the URL that I changed as explained before, this is the same > > configuration that we have been using for a long time either between > > VMs, or between docker containers in compose, or between containers > > and VM... We've never observed this kind of errors before, and our K8s > > experts do not explain what my be different in the cluster environment > > that could be the origin of the issue... > > > > More ideas from similar experiences ? > > > > Thanks again. > > > > Regards. > > > > > > > > > > Le ven. 7 juil. 2023 à 18:21, Matt Pavlovich <mattr...@gmail.com> a écrit : > >> > >> Hi Ephermeris- > >> > >> Recommendations when running in the cloud is to be sure to enable a lot of > >> self-healing features. Networking (esp in kubernetes) can be inconsistent > >> when compared to local developer desktop. > >> > >> 1. Use failover:( .. ) transport in client urls > >> 2. Use PooledConnectionFactory (esp w/ Camel) > >> 3. Configure PooledConnectionFactory to expire connections > >> 4. If possible, add a scheduled task to your Camel routes to periodically > >> restart these routes (this releases connections and allows new connections > >> to spin up) > >> 5. Look into using camel-sjms vs the Spring JMS-based component to see if > >> its a better fit > >> > >> Thanks, > >> -Matt > >> > >>> On Jul 7, 2023, at 10:47 AM, Ephemeris Lappis > >>> <ephemeris.lap...@gmail.com> wrote: > >>> > >>> Hello. > >>> > >>> We observe strange messages in our logs about ActiveMQ consumers. > >>> > >>> 17:33:29.684 WARN [Camel (XXXX_context) thread #452 - > >>> JmsConsumer[xxxx.internal.queue]] Setup of JMS message listener > >>> invoker failed for destination 'xxxx.internal.queue' - trying to > >>> recover. Cause: The Consumer is closed > >>> > >>> This message comes from Camel routes executed in Karaf. Similar > >>> messages are produced by consumers in SpringBoot applications. > >>> > >>> These WARN messages seem not to be always related to real issues, but > >>> sometimes application activities have failed after these logs > >>> appeared, and a restart of the clients have been required to restore a > >>> correct behavior. > >>> > >>> We have only seen that on a full Kubernetes configuration where both > >>> the ActiveMQ server and its clients (Camel/Kafaf or SpringBoot > >>> applications) run in the same namespace as distinct pods using a k8s > >>> service to join the openwire endpoint. > >>> > >>> Similar configurations with ActiveMQ or Karaf executed in docker > >>> containers or VM have never led to messages of this kind. > >>> > >>> So, what is the real meaning of this message whose conditions I can't > >>> identify ? > >>> > >>> Did someone already experience this behavior, in Kubernetes or not, ? > >>> > >>> Thanks in advance. > >>> > >>> Regards. > >> >
