Hi Steve,
I suggest to check the indexed data with:

curl -XGET 'http://127.0.0.1:9200/graylog_59/_search?pretty=1'

Your ES state it's yellow because you have replica unassigned, should be 
green.
Edit the config file elasticsearch.yml and disable replication :
index.number_of_replicas: 0

or configure this properly, depends of your needs.

Restart ES service.

Now the first command it's to check if your indexed data it's present in 
that indice, if yes, like I guess, You can restart a graylog and check 
again.
However now in your graylog web UI the status is it yellow or green ?
Sometimes in early versions of graylog had same problems with refresh 
between ES and web UI.
HTH 
Alberto



On Tuesday, November 10, 2015 at 7:07:36 PM UTC+1, Steve Kirkpatrick wrote:
>
> Thanks for the reply Alberto.
>
> Here's the first part of the output on the first command:
> Note: I used the IP of the graylog-server because localhost gave me: 
> Connection refused
>
> root@graylog-server:/var/log/graylog/web# curl -XGET 
> 172.20.39.61:9200/graylog_59/_stats?pretty
> {
>   "_shards" : {
>     "total" : 8,
>     "successful" : 4,
>     "failed" : 0
>   },
>   "_all" : {
>     "primaries" : {
>       "docs" : {
>         "count" : 1318398,
>         "deleted" : 0
>       },
>       "store" : {
>         "size_in_bytes" : 685371590,
>         "throttle_time_in_millis" : 0
>       },
>
> That looks promising.  Not sure if the rest of the output would be helpful.
>
> The second command:
>
> root@graylog-server:/var/log/graylog/web#  curl -XGET 
> http://172.20.39.61:9200/_cat/shards
> ...
> graylog_59 0 p STARTED    326183 161.9mb 127.0.1.1 X-Cutioner 
> graylog_59 0 r UNASSIGNED                                     
> graylog_59 3 p STARTED    329200 163.4mb 127.0.1.1 X-Cutioner 
> graylog_59 3 r UNASSIGNED                                     
> graylog_59 1 p STARTED    330482 163.4mb 127.0.1.1 X-Cutioner 
> graylog_59 1 r UNASSIGNED                                     
> graylog_59 2 p STARTED    332533 164.6mb 127.0.1.1 X-Cutioner 
> graylog_59 2 r UNASSIGNED
> ...
> graylog_67 0 p STARTED    295826 145.3mb 127.0.1.1 X-Cutioner 
> graylog_67 0 r UNASSIGNED                                     
> graylog_67 3 p STARTED    299222 146.8mb 127.0.1.1 X-Cutioner 
> graylog_67 3 r UNASSIGNED                                     
> graylog_67 1 p STARTED    298980 146.4mb 127.0.1.1 X-Cutioner 
> graylog_67 1 r UNASSIGNED                                     
> graylog_67 2 p STARTED    304105 148.4mb 127.0.1.1 X-Cutioner 
> graylog_67 2 r UNASSIGNED  
>
> Those messages seem to match the ones from the "working" indices.
>
> Third command:
>
> root@graylog-server:/var/log/graylog/web# curl -XGET 
> http://172.20.39.61:9200/_cluster/health?pretty
> {
>   "cluster_name" : "graylog2",
>   "status" : "yellow",
>   "timed_out" : false,
>   "number_of_nodes" : 2,
>   "number_of_data_nodes" : 1,
>   "active_primary_shards" : 132,
>   "active_shards" : 132,
>   "relocating_shards" : 0,
>   "initializing_shards" : 0,
>   "unassigned_shards" : 132,
>   "delayed_unassigned_shards" : 0,
>   "number_of_pending_tasks" : 0,
>   "number_of_in_flight_fetch" : 0
> }
>
> Seems OK.
>
> Any other commands I could try or logs I should look at to determine why 
> those two indices are not available within the Graylog Web UI?
>
>
> Appreciate the help.
>
> Steve.
>
> On Tuesday, November 10, 2015 at 8:19:57 AM UTC-8, Alberto Frosi wrote:
>>
>> Hi Steve,
>> I suggest to check if these indices exist yet, querying ES directly:
>>
>> curl -XGET localhost:9200/graylog_59/_stats?pretty
>>
>>  curl -XGET http://localhost:9200/_cat/shards
>>
>>
>> curl -XGET http:///localhost:9200/_cluster/health?pretty
>>
>> HTH
>> Ciao
>> Alberto
>>
>> On Tuesday, November 10, 2015 at 1:16:52 AM UTC+1, Steve Kirkpatrick 
>> wrote:
>>>
>>> Hello,
>>>
>>> Running Graylog V1.2.2 using the VM appliance from graylog.org.
>>>
>>> Been having performance issues.  When I first start Graylog, everything 
>>> is snappy.  By the next day, things have gotten more sluggish.  Sometimes 
>>> it takes 5-10 attempts to login to the web interface.  
>>>
>>> One problem I have is that two of the indices have dropped off the list 
>>> on the Systems->Indices page.
>>> After some googling, I decided to try Maintenance->Recalculate index 
>>> ranges.
>>> The job completes but neither of the two indices reappear in the list.
>>>
>>> I found these errors in /var/log/graylog/server/current:
>>>
>>> 2015-11-09_23:28:53.06554 INFO  [RebuildIndexRangesJob] Re-calculating 
>>> index ranges.
>>> 2015-11-09_23:28:53.06590 INFO  [SystemJobManager] Submitted SystemJob 
>>> <a3802c80-8739-11e5-8dd3-005056b859d5> 
>>> [org.graylog2.indexer.ranges.RebuildIndexRangesJob]
>>> 2015-11-09_23:28:53.12839 INFO  [MongoIndexRangeService] Calculated 
>>> range of [graylog_47] in [56ms].
>>> ...
>>> 2015-11-09_23:28:54.49844 INFO  [MongoIndexRangeService] Calculated 
>>> range of [graylog_55] in [101ms].
>>> 2015-11-09_23:28:54.81895 INFO  [MongoIndexRangeService] Calculated 
>>> range of [graylog_58] in [211ms].
>>> 2015-11-09_23:28:54.94361 INFO  [MongoIndexRangeService] Calculated 
>>> range of [graylog_57] in [123ms].
>>> 2015-11-09_23:28:55.04214 ERROR [Indices] Error while calculating 
>>> timestamp stats in index <graylog_59>
>>> 2015-11-09_23:28:55.04216 
>>> org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to 
>>> execute phase [query], all shards failed; shardFailures 
>>> {[XnEo6hwLTeyUZ4EluxaIEw][graylog_59][0]: 
>>> RemoteTransportException[[X-Cutioner][inet
>>> [/172.20.39.61:9300]][indices:data/read/search[phase/query]]]; nested: 
>>> ClassCastException; }{[XnEo6hwLTeyUZ4EluxaIEw][graylog_59][1]: 
>>> RemoteTransportException[[X-Cutioner][inet[/172.20.39.61:9300]][indices:data/read/search[phase/query]]];
>>>  
>>> nested: ClassCastException; }{[XnEo6hwLTeyUZ4EluxaIEw][graylog_59][2]: 
>>> RemoteTransportException[[X-Cutioner][inet[/172.20.39.61:9300]][indices:data/read/search[phase/query]]];
>>>  
>>> nested: ClassCastException; }{[XnEo6hwLTeyUZ4EluxaIEw][graylog_
>>> 59][3]: 
>>> RemoteTransportException[[X-Cutioner][inet[/172.20.39.61:9300]][indices:data/read/search[phase/query]]];
>>>  
>>> nested: ClassCastException; }
>>> 2015-11-09_23:28:55.04217       at 
>>> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:237)
>>> 2015-11-09_23:28:55.04218       at 
>>> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:183)
>>> 2015-11-09_23:28:55.04218       at 
>>> org.elasticsearch.search.action.SearchServiceTransportAction$6.handleException(SearchServiceTransportAction.java:249)
>>> 2015-11-09_23:28:55.04219       at 
>>> org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:190)
>>> 2015-11-09_23:28:55.04219       at 
>>> org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:180)
>>> 2015-11-09_23:28:55.04220       at 
>>> org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:130)
>>> 2015-11-09_23:28:55.04220       at 
>>> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>>> 2015-11-09_23:28:55.04220       at 
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> 2015-11-09_23:28:55.04221       at 
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>>> 2015-11-09_23:28:55.04221       at 
>>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>> 2015-11-09_23:28:55.04222       at 
>>> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
>>> 2015-11-09_23:28:55.04222       at 
>>> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
>>> 2015-11-09_23:28:55.04223       at 
>>> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
>>> 2015-11-09_23:28:55.04223       at 
>>> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>>> 2015-11-09_23:28:55.04224       at 
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> 2015-11-09_23:28:55.04225       at 
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>> 2015-11-09_23:28:55.04225       at 
>>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>> 2015-11-09_23:28:55.04226       at 
>>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>> 2015-11-09_23:28:55.04226       at 
>>> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
>>> 2015-11-09_23:28:55.04226       at 
>>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
>>> 2015-11-09_23:28:55.04227       at 
>>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
>>> 2015-11-09_23:28:55.04228       at 
>>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
>>> 2015-11-09_23:28:55.04228       at 
>>> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> 2015-11-09_23:28:55.04228       at 
>>> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>> 2015-11-09_23:28:55.04229       at 
>>> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> 2015-11-09_23:28:55.04229       at 
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 2015-11-09_23:28:55.04230       at 
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 2015-11-09_23:28:55.04230       at java.lang.Thread.run(Thread.java:745)
>>> 2015-11-09_23:28:55.04250 INFO  [RebuildIndexRangesJob] Could not 
>>> calculate range of index [graylog_59]. Skipping.
>>> 2015-11-09_23:28:55.04252 
>>> org.elasticsearch.indices.IndexMissingException: [graylog_59] missing
>>> 2015-11-09_23:28:55.04252       at 
>>> org.graylog2.indexer.indices.Indices.timestampStatsOfIndex(Indices.java:482)
>>> 2015-11-09_23:28:55.04253       at 
>>> org.graylog2.indexer.ranges.MongoIndexRangeService.calculateRange(MongoIndexRangeService.java:118)
>>> 2015-11-09_23:28:55.04253       at 
>>> org.graylog2.indexer.ranges.RebuildIndexRangesJob.execute(RebuildIndexRangesJob.java:96)
>>> 2015-11-09_23:28:55.04253       at 
>>> org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:88)
>>> 2015-11-09_23:28:55.04254       at 
>>> com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235)
>>> 2015-11-09_23:28:55.04254       at 
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> 2015-11-09_23:28:55.04254       at 
>>> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> 2015-11-09_23:28:55.04255       at 
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> 2015-11-09_23:28:55.04255       at 
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> 2015-11-09_23:28:55.04256       at 
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> 2015-11-09_23:28:55.04256       at 
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> 2015-11-09_23:28:55.04257       at java.lang.Thread.run(Thread.java:745)
>>> 2015-11-09_23:28:55.20408 INFO  [MongoIndexRangeService] Calculated 
>>> range of [graylog_61] in [161ms].
>>> 2015-11-09_23:28:55.38073 INFO  [MongoIndexRangeService] Calculated 
>>> range of [graylog_60] in [175ms].
>>>
>>> graylog_59 is one of the two missing indices.
>>>
>>> Is it possible to "fix" these indices and gain access to the data 
>>> contained within them?
>>> I originally configured the system to keep 30 indices, each with 24 
>>> hours of data.
>>> Today I reconfigured that to 60 indices at 12 hours each.  Not sure if 
>>> that will help with the performance issues.
>>> If their a rule-of-thumb for index sizing?
>>>
>>> Anything else I should be looking at to figure out the performance 
>>> issues?
>>> The performance graphs for the VM look OK in vSphere; no resources 
>>> appear to be overwhelmed.
>>>
>>> Thanks for any guidance.
>>>
>>> Steve.
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/d81ede60-fda1-47f1-8159-8a0b8f274449%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to