[
https://issues.apache.org/jira/browse/USERGRID-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Russo updated USERGRID-1259:
------------------------------------
Sprint: Usergrid 36 (was: Usergrid 37)
> Re-indexing ElasticSearch entity data from Cassandra - Possible Memory Leaks
> in Usergrid
> ----------------------------------------------------------------------------------------
>
> Key: USERGRID-1259
> URL: https://issues.apache.org/jira/browse/USERGRID-1259
> Project: Usergrid
> Issue Type: Story
> Affects Versions: 2.1.0
> Reporter: Jaskaran
> Fix For: 2.2.0
>
>
> Full system re-index job (http://localhost:8080/system/index/rebuild), seems
> to stop / hang after 20-30 hours of indexing. Usergrid seems to exhaust the
> 4.5 GB of RAM. Please see logs below:
> 1. UserGrid logs (out of java heap space)
> Feb 04 17:06:32 Usergrid-2 catalina.out: 06:36:31,961 WARN
> OioServerSocketPipelineSink:83 - Failed to accept a connection.
> Feb 04 17:06:32 Usergrid-2 catalina.out: java.lang.OutOfMemoryError: Java
> heap space
> Feb 04 15:05:03 Usergrid-2 catalina.out: 04:34:25,166 WARN jvm:203 -
> [default] [gc][old][29454][2683] duration [54.2s], collections [3]/[54.3s],
> total [54.2s]/[13.2h], memory [4.4gb]->[4.4gb]/[4.4gb], all_pools {[young]
> [532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [62.5mb]->[65mb]/[66.5mb]}{[old]
> [3.8gb]->[3.8gb]/[3.8gb]}
> Feb 03 20:38:34 Usergrid-2 catalina.out: 10:08:34,616 ERROR
> AmazonAsyncEventService:361 - Failed to index message:
> 886ea9bd-708d-4bea-ab1c-844ff97c947c
> 2. ES logs (time out, removed non-data node from cluster)
> Feb 04 16:00:33 Elasticsearch elasticsearch.log: [2016-02-04
> 05:31:17,243][INFO ][cluster.service ] [Arlette Truffaut] removed
> {[default][3GuynlamR6GvdiCAhhTBmw][ip-10-0-0-237][inet[/10.0.0.237:9301]]{client=true,
> data=false},}, reason:
> zen-disco-node_failed([default][3GuynlamR6GvdiCAhhTBmw][ip-10-0-0-237][inet[/10.0.0.237:9301]]{client=true,
> data=false}), reason failed to ping, tried [3] times, each with maximum
> [30s] timeout
> Feb 04 16:00:33 Elasticsearch elasticsearch.log: [2016-02-04
> 05:31:17,256][DEBUG][action.admin.cluster.node.stats] [Arlette Truffaut]
> failed to execute on node [3GuynlamR6GvdiCAhhTBmw]
> Feb 04 16:00:33 Elasticsearch elasticsearch.log:
> org.elasticsearch.transport.NodeDisconnectedException:
> [default][inet[/10.0.0.237:9301]][cluster:monitor/nodes/stats[n]] disconnected
> Is it possible that the re-indexing code in Usergrid could have memory leaks
> and thus uses up all the java heap memory.
> Please help.
> Thanks
> Jaskaran
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
