What is your client (CQ Listener) doing? Is it storing data or holding reference to data for long (memory leak?)...Does listener has any logic that takes long time, reason it may not be catching up with incoming data...You can also think about handing the event to separate thread pools....
-Anil. On Fri, Feb 10, 2017 at 12:40 PM, Barry Oglesby <[email protected]> wrote: > Dan's question is a good one. I was thinking that the issue with the > CacheClientUpdater dying was happening a lot and causing the client threads > to spin out of control. But if you're not seeing a lot of messages like > below in your client log, thats not the case. > > [info 2017/02/10 11:07:13.124 GMT <Cache Client Updater Thread on xxx> > tid=0x22] Primary subscription endpoint xxx crashed. Scheduling recovery. > > If your not seeing a lot of message like this, then your client is right > on the edge of its resources. > > The 'java.lang.OutOfMemoryError: unable to create new native thread' means > the OS doesn't have the resources to allocate a thread. It could be several > things: > > - the physical memory on the box is exhausted > - the user process limit is reached (ulimit -u) > > What is the physical memory on your client machine? What are your JVM's > -Xms and -Xms settings? Is there enough room between the physical memory > and the JVM resources for thread stacks? Are you setting -Xss (thread stack > size)? You could reduce that to save resources. But before you do anything, > you need to figure out what the threads are. > > Setting remove-unresponsive-clients=true will remove a client whose queue > is full. If a client's queue fills up, puts into it are blocked. Which > means other client requests are blocked. The main point of setting this > property is to prevent cascading issues when a client's queue is full at > the expense of causing that client to reconnect. So, there are pros and > cons to this property. > > The maximum-message-count property limits the maximum size of the client's > queue before puts block. The default is 230,000 which is mainly to handle > bursts of activity. A client's queue should never be this full in normal > processing. > > The Cache Client Updater will try forever to recover its connection to the > server. > > The QueueManagerImpl uses a ScheduledThreadPoolExecutor with a maximum > pool size of Integer.MAX_VALUE, so the number of available threads in this > area is limited only by the JVM and OS resources. > > > Thanks, > Barry Oglesby > > > On Fri, Feb 10, 2017 at 6:27 AM, Thacker, Dharam < > [email protected]> wrote: > >> Thanks Dan & Barry! >> >> Could not reproduce this issue today but got some hints based on your >> guidance with remove-unresponsive-clients and max-message-count. >> >> Somehow logs didn't give useful information, but I will enable client >> stats and verify it once again. >> >> More could you help me to understand max-message-count's role/meaning if >> I enable remove-unresponsive-clients? >> >> More how many times thread will try to recover connection in case it >> crashes? >> >> Is there any way to increase number of threads than what comes by default >> ? (Even is that recommend?) >> >> Thanks, >> Dharam >> >> Sent with Good Work (www.blackberry.com) >> On Feb 9, 2017 11:50 PM, Dan Smith <[email protected]> wrote: >> The error says "unable to create a new native thread." Do you know how >> many threads you had in this process when it died? The client stats could >> show the you the number of threads at least. You might also have an >> hs_err_pid file that might list the threads. >> >> -Dan >> >> On Thu, Feb 9, 2017 at 9:23 AM, Barry Oglesby <[email protected]> >> wrote: >> >>> Dharam, >>> >>> This stack means that the CacheClientUpdater (which is processing the >>> messages on the client) has caught an exception, is crashing and kicking >>> off a thread to recover its connection to the server. A high number of >>> events alone wouldn't cause this unless the client's queue on the server is >>> full and remove-unresponsive-client=true in the server. Is that the >>> case? It would be interesting to see the client and server logs as well as >>> the server stats to see if we can determine why this is happening. The >>> client log may contain the exception that is causing the CacheClientUpdater >>> to crash. >>> >>> Thanks, >>> Barry Oglesby >>> >>> >>> On Thu, Feb 9, 2017 at 2:42 AM, Thacker, Dharam < >>> [email protected]> wrote: >>> >>>> Hi Team, >>>> >>>> >>>> >>>> I am facing below error while working with continuous query listener. >>>> We are receiving high number of events matching to various continuous query >>>> listeners which result into below exception. Though task every thread s >>>> doing is really very small (Not at all time consuming). >>>> >>>> >>>> >>>> Any guideline on below one? Is it like thread after finishing work not >>>> going to pool? >>>> >>>> >>>> >>>> <gfe:cq-listener-container id=*"cqListenerContainer"* cache= >>>> *"event-cache"*> >>>> >>>> <gfe:listener ref=*"eventListener"* query=*"SELECT * >>>> FROM /Event WHERE entityType=’A’ and entitySubType IN SET (‘A1’,’A2') and >>>> status='PENDING'"*/> >>>> >>>> </gfe:cq-listener-container> >>>> >>>> >>>> >>>> [http-nio-8088-exec-14] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] >>>> Servlet.service() for servlet [dispatcherServlet] in context with path [] >>>> threw exception [Handler dispatch failed; nested exception is >>>> java.lang.OutOfMemoryError: unable to create new native thread] with root >>>> cause >>>> >>>> java.lang.OutOfMemoryError: unable to create new native thread >>>> >>>> at java.lang.Thread.start0(Native Method) >>>> >>>> at java.lang.Thread.start(Thread.java:714) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPool >>>> Executor.java:950) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(Threa >>>> dPoolExecutor.java:1587) >>>> >>>> at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExec >>>> ute(ScheduledThreadPoolExecutor.java:334) >>>> >>>> at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Sc >>>> heduledThreadPoolExecutor.java:533) >>>> >>>> at org.apache.geode.cache.client.internal.QueueManagerImpl.sche >>>> duleRedundancySatisfierIfNeeded(QueueManagerImpl.java:1041) >>>> >>>> at org.apache.geode.cache.client.internal.QueueManagerImpl.chec >>>> kEndpoint(QueueManagerImpl.java:430) >>>> >>>> at org.apache.geode.internal.cache.tier.sockets.CacheClientUpda >>>> ter.processMessages(CacheClientUpdater.java:1811) >>>> >>>> at org.apache.geode.internal.cache.tier.sockets.CacheClientUpda >>>> ter.run(CacheClientUpdater.java:463) >>>> >>>> >>>> >>>> Thanks & Regards, >>>> >>>> Dharam >>>> >>>> This email is confidential and subject to important disclaimers and >>>> conditions including on offers for the purchase or sale of securities, >>>> accuracy and completeness of information, viruses, confidentiality, legal >>>> privilege, and legal entity disclaimers, available at >>>> http://www.jpmorgan.com/pages/disclosures/email >>>> >>> >>> >> This email is confidential and subject to important disclaimers and >> conditions including on offers for the purchase or sale of securities, >> accuracy and completeness of information, viruses, confidentiality, legal >> privilege, and legal entity disclaimers, available at >> http://www.jpmorgan.com/pages/disclosures/email >> > >
