Jaskaran created USERGRID-1346:
----------------------------------
Summary: Tomcat - out of memory exceptions
Key: USERGRID-1346
URL: https://issues.apache.org/jira/browse/USERGRID-1346
Project: Usergrid
Issue Type: Bug
Components: Stack
Affects Versions: 2.2.0
Environment: Environment: Ubuntu 14.04, Tomcat 7, JDK 1.8.0_65
(Oracle);
Cassandra version: 2.2.6 (DataStax);
Usergrid version: 2.2.0 (Master branch, 3rd May, 2016)
Reporter: Jaskaran
Priority: Critical
Fix For: 2.2.0
Hello Usergrid Team,
We are suddenly facing "out of memory" exceptions in our Tomcat Severs, under
low load conditions. Please note, our usergrid installations have been very
stable over the last 6 months, and we have "not" seen such issues before. I am
pasting a few logs that have suddenly started showing up.
------------------------------------------------------------
Nov 09 16:15:26 catalina.out: 05:45:26,812 WARN EntityMappingParser:116 -
Encountered 2 collections consecutively. N+1 dimensional arrays are
unsupported, only arrays of depth 1 are supported
------------------------------------------------------------
Nov 09 17:22:12 catalina.out: 06:52:12,848 WARN AsyncEventServiceImpl:362 - No
index operation messages came back from event processing for msg:
------------------------------------------------------------
Nov 09 17:39:56 catalina.out: 07:09:56,177 INFO transport:470 -
[ip-10-0-2-128] failed to get local cluster state for
[#transport#-3][ip-10-0-2-128][inet[/10.0.4.205:9300]], disconnecting...
Nov 09 17:39:56 catalina.out:
org.elasticsearch.transport.ReceiveTimeoutTransportException:
[][inet[/10.0.4.205:9300]][cluster:monitor/state] request_id [11652] timed out
after [5247ms]
Nov 09 17:39:56 catalina.out: at
org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:529)
Nov 09 17:39:56 catalina.out: at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Nov 09 17:39:56 catalina.out: at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Nov 09 17:39:56 catalina.out: at java.lang.Thread.run(Thread.java:745)
------------------------------------------------------------
Nov 09 17:40:17 catalina.out: 07:10:17,557 WARN transport:415 -
[ip-10-0-2-128] Received response for a request that has timed out, sent
[10887ms] ago, timed out [3ms] ago, action [cluster:monitor/state], node
[[bluedls__us-east-1a__db__10.0.4.63][T6OWiR1US9m5ABxHh0tW0w][ip-10-0-4-63][inet[/10.0.4.63:9300]]{zone=us-east-1__us-east-1a}],
id [11678]
------------------------------------------------------------
Nov 09 17:43:05 catalina.out: 07:13:05,091 ERROR AbstractExceptionMapper:74 -
com.netflix.hystrix.exception.HystrixRuntimeException 5XX Uncaught Exception
(500)
Nov 09 17:43:05 catalina.out:
com.netflix.hystrix.exception.HystrixRuntimeException: ConsistentReplayCommand
timed-out and fallback failed.
..
Nov 09 17:43:05 catalina.out: Caused by: java.util.concurrent.TimeoutException
..
Nov 09 17:43:05 catalina.out: Caused by:
rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext
value:
org.apache.usergrid.persistence.collection.mvcc.stage.CollectionIoEvent.class
..
Nov 09 17:43:05 catalina.out: 07:13:05,123 ERROR AbstractExceptionMapper:108 -
Server Error (500):
Nov 09 17:43:05 catalina.out:
{"error":"hystrix_runtime","timestamp":1510229585122,"duration":0,"error_description":"ConsistentReplayCommand
timed-out and fallback
failed.","exception":"com.netflix.hystrix.exception.HystrixRuntimeException"}
------------------------------------------------------------
Our monitoring indicates there is no issue in cassandra and elasticseach
clusters. Look forward to your help.
Thanks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
