All settings except from ES_HEAP (set to 10 GB) are defaults, so I actually
am not sure about the new gen setting.
The host has 80 GB memory in total and 24 CPU cores. All ES indices
together sum up to ~32 GB where the biggest indices are of size ~8 GB.
We are using queries mostly together with filters and also aggregations.
We "solved" the problem with a restart of the cluster. Are there any
recommended diagnostics to be performed when this problem occurs next time?
Regards,
Abid
Am Dienstag, 24. März 2015 15:24:43 UTC+1 schrieb Jason Wee:
>
> what is the new gen setting? how much is the system memory in total?
> how many cpu cores in the node? what query do you use?
>
> jason
>
> On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain
> <hus...@novacom.mygbiz.com <javascript:>> wrote:
> > Hi all,
> >
> > we today got (for the first time) warning messages which seem to
> indicate a
> > memory problem:
> > [2015-03-24 09:08:12,960][WARN ][monitor.jvm ] [Danger]
> > [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], total
> > [5m]/[16.7m], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young]
> > [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old]
> > [6.9gb]->[3.7gb]/[8.5gb]}
> > [2015-03-24 09:08:12,960][WARN ][monitor.jvm ] [Danger]
> > [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], total
> > [18.4s]/[58s], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young]
> > [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old]
> > [6.9gb]->[3.7gb]/[8.5gb]}
> > [2015-03-24 09:08:15,372][WARN ][monitor.jvm ] [Danger]
> > [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s],
> total
> > [1.4s]/[16.7m], memory [3.7gb]->[5gb]/[9.8gb], all_pools {[young]
> > [6.1mb]->[2.7mb]/[1.1gb]}{[survivor] [0b]->[149.7mb]/[149.7mb]}{[old]
> > [3.7gb]->[4.9gb]/[8.5gb]}
> > [2015-03-24 09:08:18,192][WARN ][monitor.jvm ] [Danger]
> > [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s],
> total
> > [1.4s]/[16.7m], memory [5.8gb]->[6.2gb]/[9.8gb], all_pools {[young]
> > [845.4mb]->[1.2mb]/[1.1gb]}{[survivor]
> [149.7mb]->[149.7mb]/[149.7mb]}{[old]
> > [4.9gb]->[6gb]/[8.5gb]}
> > [2015-03-24 09:08:21,506][WARN ][monitor.jvm ] [Danger]
> > [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s],
> total
> > [1.2s]/[16.7m], memory [7gb]->[7.3gb]/[9.8gb], all_pools {[young]
> > [848.6mb]->[2.1mb]/[1.1gb]}{[survivor]
> [149.7mb]->[149.7mb]/[149.7mb]}{[old]
> > [6gb]->[7.2gb]/[8.5gb]}
> >
> > We're using ES 1.4.2 as a single node cluster, ES_HEAP is set to 10g,
> other
> > settings are defaults. From previous posts related to this issue, it is
> said
> > that field data cache may be a problem.
> >
> > Requesting /_nodes/stats/indices/fielddata says:
> > {
> > "cluster_name": "my_cluster",
> > "nodes": {
> > "ILUggMfTSvix8Kc0nfNVAw": {
> > "timestamp": 1427188716203,
> > "name": "Danger",
> > "transport_address": "inet[/192.168.110.91:9300]",
> > "host": "xxx",
> > "ip": [
> > "inet[/192.168.110.91:9300]",
> > "NONE"
> > ],
> > "indices": {
> > "fielddata": {
> > "memory_size_in_bytes": 64822224,
> > "evictions": 0
> > }
> > }
> > }
> > }
> > }
> >
> > Running top results in:
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > 12735 root 20 0 15.8g 10g 0 S 74 13.2 2485:26 java
> >
> > Any ideas what to do? If possible I would rather avoid increasing
> ES_HEAP as
> > there isn't that much free memory left on the host.
> >
> > Regards,
> >
> > Abid
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an
> > email to elasticsearc...@googlegroups.com <javascript:>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com.
>
>
> > For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
