Hi Jochen, At the default logging level, I don't see any messages in /var/log/graylog/server/current when I load the page. By some miracle, it did load for me one time after about 50 seconds. I triggered a range re-calc and I notice this:
2015-10-02_14:48:47.46793 INFO [EsIndexRangeService] Calculated range of [graylog_2656] in [119ms]. 2015-10-02_14:48:54.24635 INFO [EsIndexRangeService] Calculated range of [graylog_3503] in [51ms]. 2015-10-02_14:48:59.28810 INFO [EsIndexRangeService] Calculated range of [graylog_3984] in [133ms]. 2015-10-02_14:49:05.02345 INFO [EsIndexRangeService] Calculated range of [graylog_2655] in [120ms]. 2015-10-02_14:49:10.19885 INFO [EsIndexRangeService] Calculated range of [graylog_3502] in [95ms]. 2015-10-02_14:49:15.28163 INFO [EsIndexRangeService] Calculated range of [graylog_3985] in [31ms]. 2015-10-02_14:49:20.39475 INFO [EsIndexRangeService] Calculated range of [graylog_3509] in [58ms]. 2015-10-02_14:49:25.51083 INFO [EsIndexRangeService] Calculated range of [graylog_3508] in [70ms]. 2015-10-02_14:49:30.86208 INFO [EsIndexRangeService] Calculated range of [graylog_2661] in [175ms]. 2015-10-02_14:49:36.39929 INFO [EsIndexRangeService] Calculated range of [graylog_3993] in [48ms]. 2015-10-02_14:49:41.79540 INFO [EsIndexRangeService] Calculated range of [graylog_2660] in [104ms]. 2015-10-02_14:49:46.66168 INFO [EsIndexRangeService] Calculated range of [graylog_3994] in [31ms]. 2015-10-02_14:49:52.30633 INFO [EsIndexRangeService] Calculated range of [graylog_2663] in [38ms]. 2015-10-02_14:49:56.71214 INFO [EsIndexRangeService] Calculated range of [graylog_3510] in [61ms]. 2015-10-02_14:50:01.69542 INFO [EsIndexRangeService] Calculated range of [graylog_3991] in [116ms]. 2015-10-02_14:50:06.84462 INFO [EsIndexRangeService] Calculated range of [graylog_2662] in [52ms]. 2015-10-02_14:50:12.11462 INFO [EsIndexRangeService] Calculated range of [graylog_3992] in [39ms]. 2015-10-02_14:50:17.33784 INFO [EsIndexRangeService] Calculated range of [graylog_3990] in [108ms]. 2015-10-02_14:50:22.59067 INFO [EsIndexRangeService] Calculated range of [graylog_2647] in [95ms]. 2015-10-02_14:50:27.78069 INFO [EsIndexRangeService] Calculated range of [graylog_3979] in [173ms]. 2015-10-02_14:50:35.68699 INFO [EsIndexRangeService] Calculated range of [graylog_2646] in [87ms]. 2015-10-02_14:50:41.24406 INFO [EsIndexRangeService] Calculated range of [graylog_2649] in [79ms]. 2015-10-02_14:50:46.18656 INFO [EsIndexRangeService] Calculated range of [graylog_3977] in [147ms]. 2015-10-02_14:50:51.58839 INFO [EsIndexRangeService] Calculated range of [graylog_2648] in [77ms]. 2015-10-02_14:50:57.03334 INFO [EsIndexRangeService] Calculated range of [graylog_3978] in [93ms]. 2015-10-02_14:51:02.76092 INFO [EsIndexRangeService] Calculated range of [graylog_2643] in [62ms]. The amount of time spent calculating ranges seems to be much slower than older versions. Notice that while each range calculation completes in a few ms, there are about ~5+ seconds between each log entry. This may be expected throttling behavior, since the delay is almost always 5 seconds, but I'm not sure. In older versions, I remember seeing a list of range calculations fly by with no delay between them - they used to complete 1800 indices in about 1 minute, now it takes > 20 minutes. I think ranges used to be stored in Mongo and the performance issues started after the migration to ES, but I may be totally off-base there. And this may have nothing to do with the System->Indices page timeout, but maybe it's helpful info. I assume when the indices page loads, it needs to know the ranges of each index and that is probably what causes the majority of the delay in the page load. Thanks! On Friday, October 2, 2015 at 1:48:00 AM UTC-6, Jochen Schalanda wrote: > > Hi Jesse, > > do you see any corresponding errors in the logs of the Graylog server node? > > > Cheers, > Jochen > > On Friday, 2 October 2015 01:49:40 UTC+2, Jesse Skrivseth wrote: >> >> Since upgrading from 1.1.6 to 1.2.0, the System->Indices page - which >> used to load in 2-5 seconds - now takes several minutes before nginx times >> out with a 504 Gateway Timeout >> >> In /var/log/graylog/web/current, I see API calls failing to get index >> info. There are about 15 of these out of 1800 active (2gb) indices across 4 >> nodes. >> >> 2015-10-01_23:25:33.01576 [error] o.g.r.m.Index - Could not get index >> information for index [graylog_4006] >> 2015-10-01_23:25:33.01578 org.graylog2.restclient.lib.APIException: API >> call failed GET http://@x.x.61.48:12900/system/indexer/indices/graylog_4006 >> 2015-10-01_23:25:33.01578 at >> org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder.execute(ApiClientImpl.java:463) >> >> ~[org.graylog2.graylog2-rest-client--1.2.0-1.2.0.jar:na] >> 2015-10-01_23:25:33.01578 at >> org.graylog2.restclient.models.Index.loadIndexInfo(Index.java:83) >> [org.graylog2.graylog2-rest-client--1.2.0-1.2.0.jar:na] >> 2015-10-01_23:25:33.01579 at >> org.graylog2.restclient.models.Index.getInfo(Index.java:75) >> [org.graylog2.graylog2-rest-client--1.2.0-1.2.0.jar:na] >> 2015-10-01_23:25:33.01579 at >> views.html.system.indices.index$$anonfun$4.apply(index.template.scala:164) >> [graylog-web-interface.graylog-web-interface-1.2.0.jar:1.2.0] >> 2015-10-01_23:25:33.01579 at >> views.html.system.indices.index$$anonfun$4.apply(index.template.scala:139) >> [graylog-web-interface.graylog-web-interface-1.2.0.jar:1.2.0] >> 2015-10-01_23:25:33.01580 Caused by: java.lang.IllegalStateException: >> Unhandled error condition in API client >> 2015-10-01_23:25:33.01580 at >> org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder.execute(ApiClientImpl.java:463) >> >> ~[org.graylog2.graylog2-rest-client--1.2.0-1.2.0.jar:na] >> 2015-10-01_23:25:33.01581 at >> org.graylog2.restclient.models.Index.loadIndexInfo(Index.java:83) >> [org.graylog2.graylog2-rest-client--1.2.0-1.2.0.jar:na] >> 2015-10-01_23:25:33.01581 at >> org.graylog2.restclient.models.Index.getInfo(Index.java:75) >> [org.graylog2.graylog2-rest-client--1.2.0-1.2.0.jar:na] >> 2015-10-01_23:25:33.01581 at >> views.html.system.indices.index$$anonfun$4.apply(index.template.scala:164) >> [graylog-web-interface.graylog-web-interface-1.2.0.jar:1.2.0] >> 2015-10-01_23:25:33.01581 at >> views.html.system.indices.index$$anonfun$4.apply(index.template.scala:139) >> [graylog-web-interface.graylog-web-interface-1.2.0.jar:1.2.0] >> >> >> My guess is it's all related to the performance issues discussed here: >> https://github.com/Graylog2/graylog2-server/pull/1374 >> >> I upgraded to 1.2.1 without any apparent change in behavior. >> >> Is this a known issue? >> > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/0f71d875-9098-457d-805d-476bc7dcad23%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
