Re: [graylog2] Graylog stopped working

2017-01-02 Thread cypherbit 
Jochen,

thank you, I looked at the following logs:

root@graylog:/var/log/graylog/elasticsearch# nano current
  GNU nano 
2.2.6   
File: current

2017-01-02_09:16:55.57535 [2017-01-02 10:16:55,574][INFO 
][node ] [Molecule Man] version[2.3.1], pid[924], 
build[bd98092/2016-04-04T12:25:05Z]
2017-01-02_09:16:55.57604 [2017-01-02 10:16:55,576][INFO 
][node ] [Molecule Man] initializing ...
2017-01-02_09:16:56.80747 [2017-01-02 10:16:56,807][INFO 
][plugins  ] [Molecule Man] modules [reindex, 
lang-expression, lang-groovy], plugins [kopf], sites [kopf]
2017-01-02_09:16:56.84193 [2017-01-02 10:16:56,841][INFO 
][env  ] [Molecule Man] using [1] data paths, mounts 
[[/var/opt/graylog/data (/dev/sdb1)]], net usable_space [85.1gb], net 
total_space [98.3gb], spins? [possib$
2017-01-02_09:16:56.84211 [2017-01-02 10:16:56,842][INFO 
][env  ] [Molecule Man] heap size [1.7gb], compressed 
ordinary object pointers [true]
2017-01-02_09:16:56.84234 [2017-01-02 10:16:56,842][WARN 
][env  ] [Molecule Man] max file descriptors [64000] 
for elasticsearch process likely too low, consider increasing to at least 
[65536]
2017-01-02_09:17:02.18937 [2017-01-02 10:17:02,189][INFO 
][node ] [Molecule Man] initialized
2017-01-02_09:17:02.19168 [2017-01-02 10:17:02,191][INFO 
][node ] [Molecule Man] starting ...
2017-01-02_09:17:02.56976 [2017-01-02 10:17:02,569][INFO 
][transport] [Molecule Man] publish_address 
{192.168.1.22:9300}, bound_addresses {192.168.1.22:9300}
2017-01-02_09:17:02.57613 [2017-01-02 10:17:02,576][INFO 
][discovery] [Molecule Man] graylog/62ruQcNHSOahWbBEe71egw
2017-01-02_09:17:12.66122 [2017-01-02 10:17:12,661][INFO 
][cluster.service  ] [Molecule Man] new_master {Molecule 
Man}{62ruQcNHSOahWbBEe71egw}{192.168.1.22}{192.168.1.22:9300}, reason: 
zen-disco-join(elected_as_master, [0] joins rec$
2017-01-02_09:17:12.73775 [2017-01-02 10:17:12,737][INFO 
][http ] [Molecule Man] publish_address 
{192.168.1.22:9200}, bound_addresses {192.168.1.22:9200}
2017-01-02_09:17:12.73913 [2017-01-02 10:17:12,739][INFO 
][node ] [Molecule Man] started
2017-01-02_09:17:12.98417 [2017-01-02 10:17:12,984][INFO 
][gateway  ] [Molecule Man] recovered [1] indices into 
cluster_state
2017-01-02_09:17:15.92973 [2017-01-02 10:17:15,929][INFO 
][cluster.service  ] [Molecule Man] added 
{{graylog-52498cb4-349d-494a-8c6b-692fd78e3c6c}{56bjekcxQl6kwDCKKmeGuw}{192.168.1.22}{192.168.1.22:9350}{client=true,
 
data=false, mas$
2017-01-02_09:17:17.20882 [2017-01-02 10:17:17,208][INFO 
][cluster.routing.allocation] [Molecule Man] Cluster health status changed 
from [RED] to [YELLOW] (reason: [shards started [[graylog_0][0], 
[graylog_0][2], [graylog_0][2], [graylo$


root@graylog:/var/log/graylog/elasticsearch# nano graylog.log
[2016-12-30 07:41:38,399][WARN ][index.translog   ] [Slick] 
[graylog_0][0] failed to delete unreferenced translog files
java.nio.file.NoSuchFileException: 
/var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices/graylog_0/0/translog
at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at 
sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
at java.nio.file.Files.newDirectoryStream(Files.java:457)
at 
org.elasticsearch.index.translog.Translog$OnCloseRunnable.handle(Translog.java:726)
at 
org.elasticsearch.index.translog.Translog$OnCloseRunnable.handle(Translog.java:714)
at 
org.elasticsearch.index.translog.ChannelReference.closeInternal(ChannelReference.java:67)
at 
org.elasticsearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:64)
at 
org.elasticsearch.index.translog.TranslogReader.close(TranslogReader.java:143)
at 
org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:129)
at 
org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:354)
at 
org.elasticsearch.index.translog.Translog.(Translog.java:179)
at 
org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:208)
at 
org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:151)
at 
org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
at 
org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1515)
at 
org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1499)
at

Re: [graylog2] Graylog stopped working

2017-01-02 Thread cypherbit 
Jochen,

thank you, I looked at the following logs:

root@graylog:/var/log/graylog/elasticsearch# nano current
  GNU nano 
2.2.6   
File: current
2017-01-02_09:16:55.57535 [2017-01-02 10:16:55,574][INFO 
][node ] [Molecule Man] version[2.3.1], pid[924], 
build[bd98092/2016-04-04T12:25:05Z]
2017-01-02_09:16:55.57604 [2017-01-02 10:16:55,576][INFO 
][node ] [Molecule Man] initializing ...
2017-01-02_09:16:56.80747 [2017-01-02 10:16:56,807][INFO 
][plugins  ] [Molecule Man] modules [reindex, 
lang-expression, lang-groovy], plugins [kopf], sites [kopf]
2017-01-02_09:16:56.84193 [2017-01-02 10:16:56,841][INFO 
][env  ] [Molecule Man] using [1] data paths, mounts 
[[/var/opt/graylog/data (/dev/sdb1)]], net usable_space [85.1gb], net 
total_space [98.3gb], spins? [possib$
2017-01-02_09:16:56.84211 [2017-01-02 10:16:56,842][INFO 
][env  ] [Molecule Man] heap size [1.7gb], compressed 
ordinary object pointers [true]
2017-01-02_09:16:56.84234 [2017-01-02 10:16:56,842][WARN 
][env  ] [Molecule Man] max file descriptors [64000] 
for elasticsearch process likely too low, consider increasing to at least 
[65536]
2017-01-02_09:17:02.18937 [2017-01-02 10:17:02,189][INFO 
][node ] [Molecule Man] initialized
2017-01-02_09:17:02.19168 [2017-01-02 10:17:02,191][INFO 
][node ] [Molecule Man] starting ...
2017-01-02_09:17:02.56976 [2017-01-02 10:17:02,569][INFO 
][transport] [Molecule Man] publish_address 
{10.10.0.47:9300}, bound_addresses {10.10.0.47:9300}
2017-01-02_09:17:02.57613 [2017-01-02 10:17:02,576][INFO 
][discovery] [Molecule Man] graylog/62ruQcNHSOahWbBEe71egw
2017-01-02_09:17:12.66122 [2017-01-02 10:17:12,661][INFO 
][cluster.service  ] [Molecule Man] new_master {Molecule 
Man}{62ruQcNHSOahWbBEe71egw}{10.10.0.47}{10.10.0.47:9300}, reason: 
zen-disco-join(elected_as_master, [0] joins rec$
2017-01-02_09:17:12.73775 [2017-01-02 10:17:12,737][INFO 
][http ] [Molecule Man] publish_address 
{10.10.0.47:9200}, bound_addresses {10.10.0.47:9200}
2017-01-02_09:17:12.73913 [2017-01-02 10:17:12,739][INFO 
][node ] [Molecule Man] started
2017-01-02_09:17:12.98417 [2017-01-02 10:17:12,984][INFO 
][gateway  ] [Molecule Man] recovered [1] indices into 
cluster_state
2017-01-02_09:17:15.92973 [2017-01-02 10:17:15,929][INFO 
][cluster.service  ] [Molecule Man] added 
{{graylog-52498cb4-349d-494a-8c6b-692fd78e3c6c}{56bjekcxQl6kwDCKKmeGuw}{10.10.0.47}{10.10.0.47:9350}{client=true,
 
data=false, mas$
2017-01-02_09:17:17.20882 [2017-01-02 10:17:17,208][INFO 
][cluster.routing.allocation] [Molecule Man] Cluster health status changed 
from [RED] to [YELLOW] (reason: [shards started [[graylog_0][0], 
[graylog_0][2], [graylog_0][2], [graylo$


root@graylog:/var/log/graylog/elasticsearch# nano graylog.log
[2016-12-30 07:41:38,399][WARN ][index.translog   ] [Slick] 
[graylog_0][0] failed to delete unreferenced translog files
java.nio.file.NoSuchFileException: 
/var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices/graylog_0/0/translog
at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at 
sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
at java.nio.file.Files.newDirectoryStream(Files.java:457)
at 
org.elasticsearch.index.translog.Translog$OnCloseRunnable.handle(Translog.java:726)
at 
org.elasticsearch.index.translog.Translog$OnCloseRunnable.handle(Translog.java:714)
at 
org.elasticsearch.index.translog.ChannelReference.closeInternal(ChannelReference.java:67)
at 
org.elasticsearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:64)
at 
org.elasticsearch.index.translog.TranslogReader.close(TranslogReader.java:143)
at 
org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:129)
at 
org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:354)
at 
org.elasticsearch.index.translog.Translog.(Translog.java:179)
at 
org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:208)
at 
org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:151)
at 
org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
at 
org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1515)
at 
org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1499)
at