Thanks for the reply Alberto.
Here's the first part of the output on the first command:
Note: I used the IP of the graylog-server because localhost gave me:
Connection refused
root@graylog-server:/var/log/graylog/web# curl -XGET
172.20.39.61:9200/graylog_59/_stats?pretty
{
"_shards" : {
"total" : 8,
"successful" : 4,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 1318398,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 685371590,
"throttle_time_in_millis" : 0
},
That looks promising. Not sure if the rest of the output would be helpful.
The second command:
root@graylog-server:/var/log/graylog/web# curl -XGET
http://172.20.39.61:9200/_cat/shards
...
graylog_59 0 p STARTED 326183 161.9mb 127.0.1.1 X-Cutioner
graylog_59 0 r UNASSIGNED
graylog_59 3 p STARTED 329200 163.4mb 127.0.1.1 X-Cutioner
graylog_59 3 r UNASSIGNED
graylog_59 1 p STARTED 330482 163.4mb 127.0.1.1 X-Cutioner
graylog_59 1 r UNASSIGNED
graylog_59 2 p STARTED 332533 164.6mb 127.0.1.1 X-Cutioner
graylog_59 2 r UNASSIGNED
...
graylog_67 0 p STARTED 295826 145.3mb 127.0.1.1 X-Cutioner
graylog_67 0 r UNASSIGNED
graylog_67 3 p STARTED 299222 146.8mb 127.0.1.1 X-Cutioner
graylog_67 3 r UNASSIGNED
graylog_67 1 p STARTED 298980 146.4mb 127.0.1.1 X-Cutioner
graylog_67 1 r UNASSIGNED
graylog_67 2 p STARTED 304105 148.4mb 127.0.1.1 X-Cutioner
graylog_67 2 r UNASSIGNED
Those messages seem to match the ones from the "working" indices.
Third command:
root@graylog-server:/var/log/graylog/web# curl -XGET
http://172.20.39.61:9200/_cluster/health?pretty
{
"cluster_name" : "graylog2",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 132,
"active_shards" : 132,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 132,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}
Seems OK.
Any other commands I could try or logs I should look at to determine why
those two indices are not available within the Graylog Web UI?
Appreciate the help.
Steve.
On Tuesday, November 10, 2015 at 8:19:57 AM UTC-8, Alberto Frosi wrote:
>
> Hi Steve,
> I suggest to check if these indices exist yet, querying ES directly:
>
> curl -XGET localhost:9200/graylog_59/_stats?pretty
>
> curl -XGET http://localhost:9200/_cat/shards
>
>
> curl -XGET http:///localhost:9200/_cluster/health?pretty
>
> HTH
> Ciao
> Alberto
>
> On Tuesday, November 10, 2015 at 1:16:52 AM UTC+1, Steve Kirkpatrick wrote:
>>
>> Hello,
>>
>> Running Graylog V1.2.2 using the VM appliance from graylog.org.
>>
>> Been having performance issues. When I first start Graylog, everything
>> is snappy. By the next day, things have gotten more sluggish. Sometimes
>> it takes 5-10 attempts to login to the web interface.
>>
>> One problem I have is that two of the indices have dropped off the list
>> on the Systems->Indices page.
>> After some googling, I decided to try Maintenance->Recalculate index
>> ranges.
>> The job completes but neither of the two indices reappear in the list.
>>
>> I found these errors in /var/log/graylog/server/current:
>>
>> 2015-11-09_23:28:53.06554 INFO [RebuildIndexRangesJob] Re-calculating
>> index ranges.
>> 2015-11-09_23:28:53.06590 INFO [SystemJobManager] Submitted SystemJob
>> <a3802c80-8739-11e5-8dd3-005056b859d5>
>> [org.graylog2.indexer.ranges.RebuildIndexRangesJob]
>> 2015-11-09_23:28:53.12839 INFO [MongoIndexRangeService] Calculated range
>> of [graylog_47] in [56ms].
>> ...
>> 2015-11-09_23:28:54.49844 INFO [MongoIndexRangeService] Calculated range
>> of [graylog_55] in [101ms].
>> 2015-11-09_23:28:54.81895 INFO [MongoIndexRangeService] Calculated range
>> of [graylog_58] in [211ms].
>> 2015-11-09_23:28:54.94361 INFO [MongoIndexRangeService] Calculated range
>> of [graylog_57] in [123ms].
>> 2015-11-09_23:28:55.04214 ERROR [Indices] Error while calculating
>> timestamp stats in index <graylog_59>
>> 2015-11-09_23:28:55.04216
>> org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to
>> execute phase [query], all shards failed; shardFailures
>> {[XnEo6hwLTeyUZ4EluxaIEw][graylog_59][0]:
>> RemoteTransportException[[X-Cutioner][inet
>> [/172.20.39.61:9300]][indices:data/read/search[phase/query]]]; nested:
>> ClassCastException; }{[XnEo6hwLTeyUZ4EluxaIEw][graylog_59][1]:
>> RemoteTransportException[[X-Cutioner][inet[/172.20.39.61:9300]][indices:data/read/search[phase/query]]];
>>
>> nested: ClassCastException; }{[XnEo6hwLTeyUZ4EluxaIEw][graylog_59][2]:
>> RemoteTransportException[[X-Cutioner][inet[/172.20.39.61:9300]][indices:data/read/search[phase/query]]];
>>
>> nested: ClassCastException; }{[XnEo6hwLTeyUZ4EluxaIEw][graylog_
>> 59][3]:
>> RemoteTransportException[[X-Cutioner][inet[/172.20.39.61:9300]][indices:data/read/search[phase/query]]];
>>
>> nested: ClassCastException; }
>> 2015-11-09_23:28:55.04217 at
>> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:237)
>> 2015-11-09_23:28:55.04218 at
>> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:183)
>> 2015-11-09_23:28:55.04218 at
>> org.elasticsearch.search.action.SearchServiceTransportAction$6.handleException(SearchServiceTransportAction.java:249)
>> 2015-11-09_23:28:55.04219 at
>> org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:190)
>> 2015-11-09_23:28:55.04219 at
>> org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:180)
>> 2015-11-09_23:28:55.04220 at
>> org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:130)
>> 2015-11-09_23:28:55.04220 at
>> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>> 2015-11-09_23:28:55.04220 at
>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> 2015-11-09_23:28:55.04221 at
>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>> 2015-11-09_23:28:55.04221 at
>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>> 2015-11-09_23:28:55.04222 at
>> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
>> 2015-11-09_23:28:55.04222 at
>> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
>> 2015-11-09_23:28:55.04223 at
>> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
>> 2015-11-09_23:28:55.04223 at
>> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>> 2015-11-09_23:28:55.04224 at
>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> 2015-11-09_23:28:55.04225 at
>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>> 2015-11-09_23:28:55.04225 at
>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>> 2015-11-09_23:28:55.04226 at
>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>> 2015-11-09_23:28:55.04226 at
>> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
>> 2015-11-09_23:28:55.04226 at
>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
>> 2015-11-09_23:28:55.04227 at
>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
>> 2015-11-09_23:28:55.04228 at
>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
>> 2015-11-09_23:28:55.04228 at
>> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> 2015-11-09_23:28:55.04228 at
>> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> 2015-11-09_23:28:55.04229 at
>> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> 2015-11-09_23:28:55.04229 at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 2015-11-09_23:28:55.04230 at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 2015-11-09_23:28:55.04230 at java.lang.Thread.run(Thread.java:745)
>> 2015-11-09_23:28:55.04250 INFO [RebuildIndexRangesJob] Could not
>> calculate range of index [graylog_59]. Skipping.
>> 2015-11-09_23:28:55.04252
>> org.elasticsearch.indices.IndexMissingException: [graylog_59] missing
>> 2015-11-09_23:28:55.04252 at
>> org.graylog2.indexer.indices.Indices.timestampStatsOfIndex(Indices.java:482)
>> 2015-11-09_23:28:55.04253 at
>> org.graylog2.indexer.ranges.MongoIndexRangeService.calculateRange(MongoIndexRangeService.java:118)
>> 2015-11-09_23:28:55.04253 at
>> org.graylog2.indexer.ranges.RebuildIndexRangesJob.execute(RebuildIndexRangesJob.java:96)
>> 2015-11-09_23:28:55.04253 at
>> org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:88)
>> 2015-11-09_23:28:55.04254 at
>> com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235)
>> 2015-11-09_23:28:55.04254 at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> 2015-11-09_23:28:55.04254 at
>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> 2015-11-09_23:28:55.04255 at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> 2015-11-09_23:28:55.04255 at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> 2015-11-09_23:28:55.04256 at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> 2015-11-09_23:28:55.04256 at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> 2015-11-09_23:28:55.04257 at java.lang.Thread.run(Thread.java:745)
>> 2015-11-09_23:28:55.20408 INFO [MongoIndexRangeService] Calculated range
>> of [graylog_61] in [161ms].
>> 2015-11-09_23:28:55.38073 INFO [MongoIndexRangeService] Calculated range
>> of [graylog_60] in [175ms].
>>
>> graylog_59 is one of the two missing indices.
>>
>> Is it possible to "fix" these indices and gain access to the data
>> contained within them?
>> I originally configured the system to keep 30 indices, each with 24 hours
>> of data.
>> Today I reconfigured that to 60 indices at 12 hours each. Not sure if
>> that will help with the performance issues.
>> If their a rule-of-thumb for index sizing?
>>
>> Anything else I should be looking at to figure out the performance issues?
>> The performance graphs for the VM look OK in vSphere; no resources appear
>> to be overwhelmed.
>>
>> Thanks for any guidance.
>>
>> Steve.
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/97b65262-154c-45b8-8549-ced369c636f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
