Hi Jochen,
Thanks for you help. I attached one file of the configuration - The
configuration is the same in all 3 Graylog machines in our cluster (except
is_master parameter and the IP parameter).
I also attached one file of the logs, again - same logs are all around the
cluster (except the master periodical duties).
Our cluster:
3 Amazon instances, each one sized as m4.2xlarge - 8 cores and 32GB mem.
With Disks of 2.5TB.
Let me know if you need anything else!
Nitzan
On Thursday, February 2, 2017 at 6:38:57 PM UTC+2, Jochen Schalanda wrote:
>
> Hi Nitzan,
>
> please post the configuration and logs of all Graylog nodes and a
> description of your hardware.
>
> Cheers,
> Jochen
>
> On Thursday, 2 February 2017 17:18:12 UTC+1, Nitzan Haimovich wrote:
>>
>> Hi all,
>>
>> I'm getting this message (*Nodes with too long GC pauses*) on my Graylog
>> cluster. I saw many people were posting about it but not a single thread
>> with solutions for how to solve/fix/approach it.
>> I would be glad for any help.
>>
>> My cluster - 3 Graylog instances, each one with 8 cores and 16GB memory
>> (heap size is configured to be : Xms - 1GB , Xmx - 8GB).
>> If you need any more details please let me know.
>>
>> Thanks!!
>>
>> Nitzan
>>
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/942c1187-99bf-492a-b226-e91d89f99392%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
############################
# GRAYLOG CONFIGURATION FILE
############################
# If you are running more than one instances of Graylog server you have to
select one of these
# instances as master. The master will perform some periodical tasks that
non-masters won't perform.
is_master = true
# The auto-generated node ID will be stored in this file and read after
restarts. It is a good idea
# to use an absolute file path here if you are starting Graylog server from
init scripts or similar.
node_id_file = /etc/graylog/server/node-id
# You MUST set a secret to secure/pepper the stored user passwords here. Use at
least 64 characters.
# Generate one by using for example: pwgen -N 1 -s 96
password_secret = $SECRET
# You MUST specify a hash password for the root user (which you only need to
initially set up the
# system and in case you lose connectivity to your authentication backend)
# This password cannot be changed using the API or via the web interface. If
you need to change it,
# modify it in this file.
# Create one by using for example: echo -n yourpassword | shasum -a 256
# and put the resulting hash value into the following line
root_password_sha2 = $PASS_SHA
# The time zone setting of the root user. See
http://www.joda.org/joda-time/timezones.html for a list of valid time zones.
# Default is UTC
root_timezone = Asia/Jerusalem
# Set plugin directory here (relative or absolute)
plugin_dir = /usr/share/graylog-server/plugin
# REST API listen URI. Must be reachable by other Graylog server nodes if you
run a cluster.
# When using Graylog Collectors, this URI will be used to receive heartbeat
messages and must be accessible for all collectors.
rest_listen_uri = http://0.0.0.0:12900/
# REST API transport address. Defaults to the value of rest_listen_uri.
Exception: If rest_listen_uri
# is set to a wildcard IP address (0.0.0.0) the first non-loopback IPv4 system
address is used.
# If set, this will be promoted in the cluster discovery APIs, so other nodes
may try to connect on
# this address and it is used to generate URLs addressing entities in the REST
API. (see rest_listen_uri)
# You will need to define this, if your Graylog server is running behind a HTTP
proxy that is rewriting
# the scheme, host name or URI.
# This must not contain a wildcard address (0.0.0.0).
rest_transport_uri = http://$IP:12900/
# Enable CORS headers for REST API. This is necessary for JS-clients accessing
the server directly.
# If these are disabled, modern browsers will not be able to retrieve resources
from the server.
# This is enabled by default. Uncomment the next line to disable it.
rest_enable_cors = true
# Enable HTTPS support for the REST API. This secures the communication with
the REST API with
# TLS to prevent request forgery and eavesdropping. This is disabled by
default. Uncomment the
# next line to enable it.
#rest_enable_tls = true
# The X.509 certificate chain file in PEM format to use for securing the REST
API.
#rest_tls_cert_file = /etc/graylog/server/domain.crt
# The PKCS#8 private key file in PEM format to use for securing the REST API.
#rest_tls_key_file = /etc/graylog/server/domain.key
# Web interface listen URI.
# Configuring a path for the URI here effectively prefixes all URIs in the web
interface. This is a replacement
# for the application.context configuration parameter in pre-2.0 versions of
the Graylog web interface.
web_listen_uri = https://0.0.0.0:9000/
# Web interface endpoint URI. This setting can be overriden on a per-request
basis with the X-Graylog-Server-URL header.
# Default: $rest_transport_uri
web_endpoint_uri = https://$DNS/
# Enable CORS headers for the web interface. This is necessary for JS-clients
accessing the server directly.
# If these are disabled, modern browsers will not be able to retrieve resources
from the server.
web_enable_cors = true
# Enable HTTPS support for the web interface. This secures the communication of
the web browser with the web interface
# using TLS to prevent request forgery and eavesdropping.
# This is disabled by default. Uncomment the next line to enable it and see the
other related configuration settings.
web_enable_tls = true
# The X.509 certificate chain file in PEM format to use for securing the web
interface.
web_tls_cert_file = /etc/graylog/server/domain.crt
# The PKCS#8 private key file in PEM format to use for securing the web
interface.
web_tls_key_file = /etc/graylog/server/domain.key
#########################################################################
# Configuration file for the embedded Elasticsearch instance in Graylog #
#########################################################################
# Pay attention to the working directory of the server, maybe use an absolute
path here.
# Default: empty
#elasticsearch_config_file = /etc/graylog/server/elasticsearch.yml
# Graylog will use multiple indices to store documents in. You can configured
the strategy it uses to determine
# when to rotate the currently active write index.
# It supports multiple rotation strategies:
# - "count" of messages per index, use elasticsearch_max_docs_per_index below
to configure
# - "size" per index, use elasticsearch_max_size_per_index below to configure
# valid values are "count", "size" and "time", default is "count"
rotation_strategy = size
# (Approximate) maximum size in bytes per Elasticsearch index on disk before a
new index is being created, also see
# no_retention and elasticsearch_max_number_of_indices. Default is 1GB.
# Configure this if you used 'rotation_strategy = size' above.
elasticsearch_max_size_per_index = 10737418240
# How many indices do you want to keep?
elasticsearch_max_number_of_indices = 150
# Decide what happens with the oldest indices when the maximum number of
indices is reached.
# The following strategies are availble:
# - delete # Deletes the index completely (Default)
# - close # Closes the index and hides it from the system. Can be re-opened
later.
retention_strategy = close
# How many Elasticsearch shards and replicas should be used per index? Note
that this only applies to newly created indices.
elasticsearch_shards = 6
elasticsearch_replicas = 1
# Prefix for all Elasticsearch indices and index aliases managed by Graylog.
elasticsearch_index_prefix = graylog
# Do you want to allow searches with leading wildcards? This can be extremely
resource hungry and should only
# be enabled with care. See also:
http://docs.graylog.org/en/2.1/pages/queries.html
allow_leading_wildcard_searches = false
# Do you want to allow searches to be highlighted? Depending on the size of
your messages this can be memory hungry and
# should only be enabled after making sure your Elasticsearch cluster has
enough memory.
allow_highlighting = false
# settings to be passed to elasticsearch's client (overriding those in the
provided elasticsearch_config_file)
# this must be the same as for your Elasticsearch cluster
elasticsearch_cluster_name = elasticsearch-graylog
# The prefix being used to generate the Elasticsearch node name which makes it
easier to identify the specific Graylog
# server running the embedded Elasticsearch instance. The node name will be
constructed by concatenating this prefix
# and the Graylog node ID (see node_id_file), for example
"graylog-17052010-1234-5678-abcd-1337cafebabe".
# Default: graylog-
elasticsearch_node_name_prefix = graylog-
# A comma-separated list of Elasticsearch nodes which Graylog is using to
connect to the Elasticsearch cluster,
# see
https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-discovery-zen.html
for details.
# Default: 127.0.0.1
elasticsearch_discovery_zen_ping_unicast_hosts = $ES_DNS:9300
# use a different port if you run multiple Elasticsearch nodes on one machine
elasticsearch_transport_tcp_port = 9350
# Change the following setting if you are running into problems with timeouts
during Elasticsearch cluster discovery.
# The setting is specified in milliseconds, the default is 5000ms (5 seconds).
#elasticsearch_cluster_discovery_timeout = 5000
# the following settings allow to change the bind addresses for the
Elasticsearch client in Graylog
# these settings are empty by default, letting Elasticsearch choose
automatically,
# override them here or in the 'elasticsearch_config_file' if you need to bind
to a special address
# refer to
https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-network.html
# for special values here
elasticsearch_network_host = 0.0.0.0
#elasticsearch_network_bind_host =
#elasticsearch_network_publish_host =
# Analyzer (tokenizer) to use for message and full_message field. The
"standard" filter usually is a good idea.
# All supported analyzers are: standard, simple, whitespace, stop, keyword,
pattern, language, snowball, custom
# Elasticsearch documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/2.3/analysis.html
# Note that this setting only takes effect on newly created indices.
elasticsearch_analyzer = standard
# Global request timeout for Elasticsearch requests (e. g. during search, index
creation, or index time-range
# calculations) based on a best-effort to restrict the runtime of Elasticsearch
operations.
# Default: 1m
#elasticsearch_request_timeout = 1m
# Time interval for index range information cleanups. This setting defines how
often stale index range information
# is being purged from the database.
# Default: 1h
#index_ranges_cleanup_interval = 1h
# Batch size for the Elasticsearch output. This is the maximum (!) number of
messages the Elasticsearch output
# module will get at once and write to Elasticsearch in a batch call. If the
configured batch size has not been
# reached within output_flush_interval seconds, everything that is available
will be flushed at once. Remember
# that every outputbuffer processor manages its own batch and performs its own
batch write calls.
# ("outputbuffer_processors" variable)
output_batch_size = 40000
# Flush interval (in seconds) for the Elasticsearch output. This is the maximum
amount of time between two
# batches of messages written to Elasticsearch. It is only effective at all if
your minimum number of messages
# for this time period is less than output_batch_size * outputbuffer_processors.
output_flush_interval = 1
# As stream outputs are loaded only on demand, an output which is failing to
initialize will be tried over and
# over again. To prevent this, the following configuration options define after
how many faults an output will
# not be tried again for an also configurable amount of seconds.
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
# The number of parallel running processors.
# Raise this number if your buffers are filling up.
processbuffer_processors = 30
outputbuffer_processors = 50
#outputbuffer_processor_keep_alive_time = 5000
#outputbuffer_processor_threads_core_pool_size = 3
#outputbuffer_processor_threads_max_pool_size = 30
# UDP receive buffer size for all message inputs (e. g. SyslogUDPInput).
#udp_recvbuffer_sizes = 1048576
# Wait strategy describing how buffer processors wait on a cursor sequence.
(default: sleeping)
# Possible types:
# - yielding
# Compromise between performance and CPU usage.
# - sleeping
# Compromise between performance and CPU usage. Latency spikes can occur
after quiet periods.
# - blocking
# High throughput, low latency, higher CPU usage.
# - busy_spinning
# Avoids syscalls which could introduce latency jitter. Best when threads
can be bound to specific CPU cores.
processor_wait_strategy = blocking
# Size of internal ring buffers. Raise this if raising outputbuffer_processors
does not help anymore.
# For optimum performance your LogMessage objects in the ring buffer should fit
in your CPU L3 cache.
# Must be a power of 2. (512, 1024, 2048, ...)
ring_size = 262144
inputbuffer_ring_size = 131072
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
# Enable the disk based message journal.
message_journal_enabled = true
# The directory which will be used to store the message journal. The directory
must me exclusively used by Graylog and
# must not contain any other files than the ones created by Graylog itself.
#
# ATTENTION:
# If you create a seperate partition for the journal files and use a file
system creating directories like 'lost+found'
# in the root directory, you need to create a sub directory for your journal.
# Otherwise Graylog will log an error message that the journal is corrupt and
Graylog will not start.
message_journal_dir = /var/lib/graylog-server/journal
# Journal hold messages before they could be written to Elasticsearch.
# For a maximum of 12 hours or 5 GB whichever happens first.
# During normal operation the journal will be smaller.
#message_journal_max_age = 12h
#message_journal_max_size = 10gb
#message_journal_flush_age = 30s
#message_journal_flush_interval = 18000
#message_journal_segment_age = 1h
#message_journal_segment_size = 800mb
# Number of threads used exclusively for dispatching internal events. Default
is 2.
#async_eventbus_processors = 2
# How many seconds to wait between marking node as DEAD for possible load
balancers and starting the actual
# shutdown process. Set to 0 if you have no status checking load balancers in
front.
lb_recognition_period_seconds = 3
# Journal usage percentage that triggers requesting throttling for this server
node from load balancers. The feature is
# disabled if not set.
lb_throttle_threshold_percentage = 98
# Every message is matched against the configured streams and it can happen
that a stream contains rules which
# take an unusual amount of time to run, for example if its using regular
expressions that perform excessive backtracking.
# This will impact the processing of the entire server. To keep such
misbehaving stream rules from impacting other
# streams, Graylog limits the execution time for each stream.
# The default values are noted below, the timeout is in milliseconds.
# If the stream matching for one stream took longer than the timeout value, and
this happened more than "max_faults" times
# that stream is disabled and a notification is shown in the web interface.
stream_processing_timeout = 5000
stream_processing_max_faults = 0
# Length of the interval in seconds in which the alert conditions for all
streams should be checked
# and alarms are being sent.
#alert_check_interval = 60
# Since 0.21 the Graylog server supports pluggable output modules. This means a
single message can be written to multiple
# outputs. The next setting defines the timeout for a single output module,
including the default output module where all
# messages end up.
#
# Time in milliseconds to wait for all message outputs to finish writing a
single message.
#output_module_timeout = 10000
# Time in milliseconds after which a detected stale master node is being
rechecked on startup.
#stale_master_timeout = 2000
# Time in milliseconds which Graylog is waiting for all threads to stop on
shutdown.
#shutdown_timeout = 30000
# MongoDB connection string for single instance
#mongodb_uri =
# MongoDB connection string for replica set
mongodb_uri =
mongodb://$MONGO_USER:$MONGO_PASS@$MONGO_CLUSTER/$MONGO_DB_NAME?replicaSet=$REPLICA_SET
# Increase this value according to the maximum connections your MongoDB server
can handle from a single client
# if you encounter MongoDB connection problems.
mongodb_max_connections = 1000
# Number of threads allowed to be blocked by MongoDB connections multiplier.
Default: 5
# If mongodb_max_connections is 100, and
mongodb_threads_allowed_to_block_multiplier is 5,
# then 500 threads can block. More than that and an exception will be thrown.
#
http://api.mongodb.com/java/current/com/mongodb/MongoOptions.html#threadsAllowedToBlockForConnectionMultiplier
mongodb_threads_allowed_to_block_multiplier = 5
# Drools Rule File (Use to rewrite incoming log messages)
# See: http://docs.graylog.org/en/2.1/pages/drools.html
rules_file = /etc/graylog/server/rules.drl
# The default connect timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when
converted to milliseconds).
# Default: 5s
#http_connect_timeout = 5s
# The default read timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when
converted to milliseconds).
# Default: 10s
#http_read_timeout = 10s
# The default write timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when
converted to milliseconds).
# Default: 10s
#http_write_timeout = 10s
# HTTP proxy for outgoing HTTP connections
#http_proxy_uri =
# The threshold of the garbage collection runs. If GC runs take longer than
this threshold, a system notification
# will be generated to warn the administrator about possible problems with the
system. Default is 1 second.
#gc_warning_threshold = 1s
# The default cache time for dashboard widgets. (Default: 10 seconds, minimum:
1 second)
#dashboard_widget_default_cache_time = 10s
# Automatically load content packs in "content_packs_dir" on the first start of
Graylog.
content_packs_loader_enabled = true
# The directory which contains content packs which should be loaded on the
first start of Graylog.
content_packs_dir = /usr/share/graylog-server/contentpacks
# A comma-separated list of content packs (files in "content_packs_dir") which
should be applied on
# the first start of Graylog.
# Default: empty
content_packs_auto_load = grok-patterns.json
2017-02-01T06:04:33.165Z INFO [CmdLineTool] Loaded plugin:
OpsGenieAlarmCallback 1.0.0
[com.opsgenie.plugin.graylog.OpsGenieAlarmCallbackPlugin]
2017-02-01T06:04:33.167Z INFO [CmdLineTool] Loaded plugin: Elastic Beats Input
2.2.0-beta.6 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-02-01T06:04:33.168Z INFO [CmdLineTool] Loaded plugin: Collector
2.2.0-beta.6 [org.graylog.plugins.collector.CollectorPlugin]
2017-02-01T06:04:33.169Z INFO [CmdLineTool] Loaded plugin: Enterprise
Integration Plugin 2.2.0-beta.6
[org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-02-01T06:04:33.171Z INFO [CmdLineTool] Loaded plugin: MapWidgetPlugin
2.2.0-beta.6 [org.graylog.plugins.map.MapWidgetPlugin]
2017-02-01T06:04:33.172Z INFO [CmdLineTool] Loaded plugin: Pipeline Processor
Plugin 2.2.0-beta.6 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-02-01T06:04:33.173Z INFO [CmdLineTool] Loaded plugin: Anonymous Usage
Statistics 2.2.0-beta.6 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-02-01T06:04:33.174Z INFO [CmdLineTool] Loaded plugin: Slack
2.2.0-SNAPSHOT [org.graylog2.plugins.slack.callback.SlackAlarmCallback]
2017-02-01T06:04:33.374Z INFO [CmdLineTool] Running with JVM arguments: -Xms1g
-Xmx16g -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow
-Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml
-Djava.library.path=/usr/share/graylog-server/lib/sigar
-Dgraylog2.installation_source=deb
2017-02-01T06:04:33.566Z INFO [Version] HV000001: Hibernate Validator null
2017-02-01T06:04:35.744Z INFO [InputBufferImpl] Message journal is enabled.
2017-02-01T06:04:35.770Z INFO [NodeId] Node ID:
f7636182-5b01-4fb6-965d-b904415109aa
2017-02-01T06:04:36.034Z INFO [LogManager] Loading logs.
2017-02-01T06:04:36.038Z INFO [LogManager] Logs loading complete.
2017-02-01T06:04:36.070Z INFO [LogManager] Created log for partition
[messagejournal,0] in /var/lib/graylog-server/journal with properties
{file.delete.delay.ms -> 60000, compact -> false, max.message.bytes ->
104857600, min.insync.replicas -> 1, segment.jitter.ms -> 0,
index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5,
unclean.leader.election.enable -> true, retention.bytes -> 5368709120,
delete.retention.ms -> 86400000, flush.ms -> 60000, segment.bytes -> 104857600,
segment.ms -> 3600000, retention.ms -> 43200000, flush.messages -> 1000000,
segment.index.bytes -> 1048576}.
2017-02-01T06:04:36.070Z INFO [KafkaJournal] Initialized Kafka based journal
at /var/lib/graylog-server/journal
2017-02-01T06:04:36.081Z INFO [InputBufferImpl] Initialized InputBufferImpl
with ring size <131072> and wait strategy <BlockingWaitStrategy>, running 2
parallel message handlers.
2017-02-01T06:04:36.095Z INFO [cluster] Cluster created with settings
{hosts=[IP1:27017, IP2:27017, IP3:27017], mode=MULTIPLE,
requiredClusterType=REPLICA_SET, serverSelectionTimeout='30000 ms',
maxWaitQueueSize=5000, requiredReplicaSetName='REPLICA_NAME'}
2017-02-01T06:04:36.096Z INFO [cluster] Adding discovered server IP1:27017 to
client view of cluster
2017-02-01T06:04:36.119Z INFO [cluster] Adding discovered server IP2:27017 to
client view of cluster
2017-02-01T06:04:36.120Z INFO [cluster] Adding discovered server IP3:27017 to
client view of cluster
2017-02-01T06:04:36.143Z INFO [cluster] No server chosen by
ReadPreferenceServerSelector{readPreference=primary} from cluster description
ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE,
serverDescriptions=[ServerDescription{address=IP1:27017, type=UNKNOWN,
state=CONNECTING}, ServerDescription{address=IP2:27017, type=UNKNOWN,
state=CONNECTING}, ServerDescription{address=IP3:27017, type=UNKNOWN,
state=CONNECTING}]}. Waiting for 30000 ms before timing out
2017-02-01T06:04:36.160Z INFO [connection] Opened connection
[connectionId{localValue:3, serverValue:38653}] to IP1:27017
2017-02-01T06:04:36.161Z INFO [connection] Opened connection
[connectionId{localValue:2, serverValue:10338}] to IP2:27017
2017-02-01T06:04:36.162Z INFO [cluster] Monitor thread successfully connected
to server with description ServerDescription...
2017-02-01T06:04:36.162Z INFO [cluster] Monitor thread successfully connected
to server with description ServerDescription...
2017-02-01T06:04:36.164Z INFO [connection] Opened connection
[connectionId{localValue:1, serverValue:43114}] to IP3:27017
2017-02-01T06:04:36.166Z INFO [cluster] Monitor thread successfully connected
to server with description ServerDescription...
2017-02-01T06:04:36.167Z INFO [cluster] Discovered replica set primary
IP3:27017
2017-02-01T06:04:36.175Z INFO [connection] Opened connection
[connectionId{localValue:4, serverValue:43115}] to IP3:27017
2017-02-01T06:04:36.389Z INFO [node]
[graylog-f7636182-5b01-4fb6-965d-b904415109aa] version[2.4.3], pid[23243],
build[d38a34e/2016-12-07T16:28:56Z]
2017-02-01T06:04:36.389Z INFO [node]
[graylog-f7636182-5b01-4fb6-965d-b904415109aa] initializing ...
2017-02-01T06:04:36.394Z INFO [plugins]
[graylog-f7636182-5b01-4fb6-965d-b904415109aa] modules [], plugins
[graylog-monitor], sites []
2017-02-01T06:04:37.601Z INFO [node]
[graylog-f7636182-5b01-4fb6-965d-b904415109aa] initialized
2017-02-01T06:04:37.685Z INFO [ProcessBuffer] Initialized ProcessBuffer with
ring size <262144> and wait strategy <BlockingWaitStrategy>.
2017-02-01T06:04:39.316Z WARN [DroolsEngine] Could not read drools source
file. Not loading rules: /etc/graylog/server/rules.drl (No such file or
directory)
2017-02-01T06:04:39.316Z WARN [RulesEngineProvider] Unable to load rules due
to load error: /etc/graylog/server/rules.drl
2017-02-01T06:04:39.557Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:39.563Z INFO [OutputBuffer] Initialized OutputBuffer with
ring size <262144> and wait strategy <BlockingWaitStrategy>.
2017-02-01T06:04:39.705Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:39.908Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:40.026Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:40.139Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:40.258Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:40.366Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:40.503Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:40.611Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:40.716Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:40.807Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:40.906Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:41.012Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:41.107Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:41.189Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:41.266Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:41.364Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:41.451Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:41.534Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:41.617Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:41.696Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:41.803Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:41.878Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:41.974Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:42.058Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:42.144Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:42.265Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:42.352Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:42.433Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:42.510Z WARN [GeoIpResolverEngine] GeoIP database file does
not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-02-01T06:04:43.420Z INFO [ServerBootstrap] Graylog server
2.2.0-beta.6+ca80b06 starting up
2017-02-01T06:04:43.420Z INFO [ServerBootstrap] JRE: Oracle Corporation
1.8.0_111 on Linux 3.13.0-48-generic
2017-02-01T06:04:43.420Z INFO [ServerBootstrap] Deployment: deb
2017-02-01T06:04:43.421Z INFO [ServerBootstrap] OS: Ubuntu 14.04.3 LTS (trusty)
2017-02-01T06:04:43.421Z INFO [ServerBootstrap] Arch: amd64
2017-02-01T06:04:43.425Z WARN [DeadEventLoggingListener] Received unhandled
event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus
<AsyncEventBus{graylog-eventbus}>
2017-02-01T06:04:43.442Z INFO [PeriodicalsService] Starting 26 periodicals ...
2017-02-01T06:04:43.442Z INFO [Periodicals] Starting
[org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling
every [1s].
2017-02-01T06:04:43.492Z INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.AlertScannerThread] periodical. Only started on
Graylog master nodes.
2017-02-01T06:04:43.493Z INFO [Periodicals] Starting
[org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in
[0s], polling every [1s].
2017-02-01T06:04:43.493Z INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.ClusterHealthCheckThread] periodical. Only started on
Graylog master nodes.
2017-02-01T06:04:43.493Z INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Only started
on Graylog master nodes.
2017-02-01T06:04:43.493Z INFO [Periodicals] Starting
[org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running
forever.
2017-02-01T06:04:43.493Z INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.IndexerClusterCheckerThread] periodical. Only started
on Graylog master nodes.
2017-02-01T06:04:43.493Z INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.IndexRetentionThread] periodical. Only started on
Graylog master nodes.
2017-02-01T06:04:43.494Z INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.IndexRotationThread] periodical. Only started on
Graylog master nodes.
2017-02-01T06:04:43.494Z INFO [Periodicals] Starting
[org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2017-02-01T06:04:43.494Z INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.VersionCheckThread] periodical. Only started on
Graylog master nodes.
2017-02-01T06:04:43.494Z INFO [Periodicals] Starting
[org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s],
polling every [1s].
2017-02-01T06:04:43.494Z INFO [Periodicals] Starting
[org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every
[1s].
2017-02-01T06:04:43.495Z INFO [PeriodicalsService] Not starting
[org.graylog2.events.ClusterEventCleanupPeriodical] periodical. Only started on
Graylog master nodes.
2017-02-01T06:04:43.495Z INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical. Only started
on Graylog master nodes.
2017-02-01T06:04:43.495Z INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical. Only
started on Graylog master nodes.
2017-02-01T06:04:43.495Z INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical. Only started
on Graylog master nodes.
2017-02-01T06:04:43.495Z INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Only
started on Graylog master nodes.
2017-02-01T06:04:43.495Z INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical. Only
started on Graylog master nodes.
2017-02-01T06:04:43.495Z INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.ConfigurationManagementPeriodical] periodical. Only
started on Graylog master nodes.
2017-02-01T06:04:43.495Z INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.LdapGroupMappingMigration] periodical. Only started on
Graylog master nodes.
2017-02-01T06:04:43.509Z INFO [node]
[graylog-f7636182-5b01-4fb6-965d-b904415109aa] starting ...
2017-02-01T06:04:43.511Z INFO [Periodicals] Starting
[org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2017-02-01T06:04:43.522Z INFO [Periodicals] Starting
[org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in
[300s], polling every [21600s].
2017-02-01T06:04:43.523Z INFO [PeriodicalsService] Not starting
[org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical.
Not configured to run on this node.
2017-02-01T06:04:43.524Z INFO [PeriodicalsService] Not starting
[org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration]
periodical. Only started on Graylog master nodes.
2017-02-01T06:04:43.525Z INFO [PeriodicalsService] Not starting
[org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread]
periodical. Only started on Graylog master nodes.
2017-02-01T06:04:43.523Z INFO [connection] Opened connection
[connectionId{localValue:6, serverValue:43118}] to IP3:27017
2017-02-01T06:04:43.530Z INFO [connection] Opened connection
[connectionId{localValue:5, serverValue:43117}] to IP3:27017
2017-02-01T06:04:43.651Z INFO [transport]
[graylog-f7636182-5b01-4fb6-965d-b904415109aa] publish_address {IP1:9350},
bound_addresses {[::]:9350}
2017-02-01T06:04:43.658Z INFO [discovery]
[graylog-f7636182-5b01-4fb6-965d-b904415109aa]
elasticsearch-graylog/6QDVOQvNRfWYvZ6iiFMLkg
2017-02-01T06:04:43.877Z INFO [JerseyService] Enabling CORS for HTTP endpoint
2017-02-01T06:04:46.664Z WARN [discovery]
[graylog-f7636182-5b01-4fb6-965d-b904415109aa] waited for 3s and no initial
state was set by the discovery
2017-02-01T06:04:46.665Z INFO [node]
[graylog-f7636182-5b01-4fb6-965d-b904415109aa] started
2017-02-01T06:04:46.737Z INFO [service]
[graylog-f7636182-5b01-4fb6-965d-b904415109aa] detected_master ....reason:
zen-disco-receive(from master ...
2017-02-01T06:04:53.355Z INFO [NetworkListener] Started listener bound to
[0.0.0.0:12900]
2017-02-01T06:04:53.357Z INFO [HttpServer] [HttpServer] Started.
2017-02-01T06:04:53.357Z INFO [JerseyService] Started REST API at
<http://0.0.0.0:12900/>
2017-02-01T06:04:53.380Z INFO [JerseyService] Enabling CORS for HTTP endpoint
2017-02-01T06:04:56.344Z INFO [NetworkListener] Started listener bound to
[0.0.0.0:9000]
2017-02-01T06:04:56.344Z INFO [HttpServer] [HttpServer-1] Started.
2017-02-01T06:04:56.344Z INFO [JerseyService] Started Web Interface at
<https://0.0.0.0:9000/>
2017-02-01T06:04:56.345Z INFO [ServiceManagerListener] Services are healthy
2017-02-01T06:04:56.345Z INFO [ServerBootstrap] Services started, startup
times in ms: {InputSetupService [RUNNING]=3, OutputSetupService [RUNNING]=77,
BufferSynchronizerService [RUNNING]=78, KafkaJournal [RUNNING]=80,
PeriodicalsService [RUNNING]=85, JournalReader [RUNNING]=85,
ConfigurationEtagService [RUNNING]=87, StreamCacheService [RUNNING]=111,
IndexerSetupService [RUNNING]=3334, JerseyService [RUNNING]=12903}
2017-02-01T06:04:56.346Z INFO [InputSetupService] Triggering launching
persisted inputs, node transitioned from Uninitialized?[LB:DEAD] to
Running?[LB:ALIVE]
2017-02-01T06:04:56.352Z INFO [ServerBootstrap] Graylog server up and running.
2017-02-01T06:04:56.357Z INFO [InputStateListener] Input [Raw/Plaintext
Kafka/588759e1cd48734b1797255c] is now STARTING
2017-02-01T06:04:56.359Z INFO [InputStateListener] Input [Raw/Plaintext
Kafka/588f5f75cd48732a8e71d7f9] is now STARTING
2017-02-01T06:04:56.360Z INFO [InputStateListener] Input [Raw/Plaintext
Kafka/588e0185cd487370d64bf67d] is now STARTING
2017-02-01T06:04:56.371Z INFO [VerifiableProperties] Verifying properties
2017-02-01T06:04:56.371Z INFO [VerifiableProperties] Verifying properties
2017-02-01T06:04:56.372Z INFO [VerifiableProperties] Verifying properties
2017-02-01T06:04:56.375Z INFO [VerifiableProperties] Property
auto.commit.interval.ms is overridden to 1000
2017-02-01T06:04:56.375Z INFO [VerifiableProperties] Property
auto.commit.interval.ms is overridden to 1000
2017-02-01T06:04:56.375Z INFO [VerifiableProperties] Property
auto.commit.interval.ms is overridden to 1000
2017-02-01T06:04:56.375Z INFO [VerifiableProperties] Property client.id is
overridden to gl2-f7636182-5b01-4fb6-965d-b904415109aa-588f5f75cd48732a8e71d7f9
2017-02-01T06:04:56.375Z INFO [VerifiableProperties] Property client.id is
overridden to gl2-f7636182-5b01-4fb6-965d-b904415109aa-588e0185cd487370d64bf67d
2017-02-01T06:04:56.375Z INFO [VerifiableProperties] Property client.id is
overridden to gl2-f7636182-5b01-4fb6-965d-b904415109aa-588759e1cd48734b1797255c
2017-02-01T06:04:56.375Z INFO [VerifiableProperties] Property
consumer.timeout.ms is overridden to 1000
2017-02-01T06:04:56.375Z INFO [VerifiableProperties] Property
consumer.timeout.ms is overridden to 1000
2017-02-01T06:04:56.375Z INFO [VerifiableProperties] Property
consumer.timeout.ms is overridden to 1000
2017-02-01T06:04:56.375Z INFO [VerifiableProperties] Property fetch.min.bytes
is overridden to 5
2017-02-01T06:04:56.375Z INFO [VerifiableProperties] Property fetch.min.bytes
is overridden to 5
2017-02-01T06:04:56.376Z INFO [VerifiableProperties] Property fetch.min.bytes
is overridden to 5
2017-02-01T06:04:56.376Z INFO [VerifiableProperties] Property
fetch.wait.max.ms is overridden to 100
2017-02-01T06:04:56.376Z INFO [VerifiableProperties] Property
fetch.wait.max.ms is overridden to 100
2017-02-01T06:04:56.376Z INFO [VerifiableProperties] Property
fetch.wait.max.ms is overridden to 100
2017-02-01T06:04:56.376Z INFO [VerifiableProperties] Property group.id is
overridden to graylog2
2017-02-01T06:04:56.376Z INFO [VerifiableProperties] Property group.id is
overridden to graylog2
2017-02-01T06:04:56.376Z INFO [VerifiableProperties] Property group.id is
overridden to graylog2
2017-02-01T06:04:56.376Z INFO [VerifiableProperties] Property
zookeeper.connect is overridden to ...
2017-02-01T06:04:56.376Z INFO [VerifiableProperties] Property
zookeeper.connect is overridden to ...
2017-02-01T06:04:56.376Z INFO [VerifiableProperties] Property
zookeeper.connect is overridden to ...
2017-02-01T06:04:56.394Z INFO [ZookeeperConsumerConnector]
[graylog2_...-1485929096393-25d999de], Connecting to zookeeper instance at ...
2017-02-01T06:04:56.394Z INFO [ZookeeperConsumerConnector]
[graylog2_...-1485929096393-dfeecc8f], Connecting to zookeeper instance at ...
2017-02-01T06:04:56.394Z INFO [ZookeeperConsumerConnector]
[graylog2_...-1485929096393-7617aa40], Connecting to zookeeper instance at ...
2017-02-01T06:04:56.403Z INFO [ZkEventThread] Starting ZkClient event thread.
2017-02-01T06:04:56.403Z INFO [ZkEventThread] Starting ZkClient event thread.
2017-02-01T06:04:56.404Z INFO [ZkEventThread] Starting ZkClient event thread.
2017-02-01T06:04:56.410Z INFO [ZooKeeper] Client
environment:zookeeper.version=3.4.9-1757313, built on 08/23/2016 06:50 GMT
2017-02-01T06:04:56.410Z INFO [ZooKeeper] Client environment:host.name=...
2017-02-01T06:04:56.410Z INFO [ZooKeeper] Client
environment:java.version=1.8.0_111
2017-02-01T06:04:56.410Z INFO [ZooKeeper] Client
environment:java.vendor=Oracle Corporation
2017-02-01T06:04:56.411Z INFO [ZooKeeper] Client
environment:java.home=/usr/lib/jvm/java-8-oracle/jre
2017-02-01T06:04:56.411Z INFO [ZooKeeper] Client
environment:java.class.path=/usr/share/graylog-server/graylog.jar
2017-02-01T06:04:56.411Z INFO [ZooKeeper] Client
environment:java.library.path=/usr/share/graylog-server/lib/sigar
2017-02-01T06:04:56.411Z INFO [ZooKeeper] Client
environment:java.io.tmpdir=/tmp
2017-02-01T06:04:56.411Z INFO [ZooKeeper] Client environment:java.compiler=<NA>
2017-02-01T06:04:56.411Z INFO [ZooKeeper] Client environment:os.name=Linux
2017-02-01T06:04:56.411Z INFO [ZooKeeper] Client environment:os.arch=amd64
2017-02-01T06:04:56.411Z INFO [ZooKeeper] Client
environment:os.version=3.13.0-48-generic
2017-02-01T06:04:56.411Z INFO [ZooKeeper] Client environment:user.name=graylog
2017-02-01T06:04:56.411Z INFO [ZooKeeper] Client
environment:user.home=/var/lib/graylog-server
2017-02-01T06:04:56.411Z INFO [ZooKeeper] Client environment:user.dir=/
2017-02-01T06:04:56.412Z INFO [ZooKeeper] Initiating client connection,
connectString=... sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@7db6d0ea
2017-02-01T06:04:56.412Z INFO [ZooKeeper] Initiating client connection,
connectString=... sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@5b1280f5
2017-02-01T06:04:56.412Z INFO [ZooKeeper] Initiating client connection,
connectString=... sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@16a80671
2017-02-01T06:04:56.424Z INFO [ZkClient] Waiting for keeper state SyncConnected
2017-02-01T06:04:56.425Z INFO [ZkClient] Waiting for keeper state SyncConnected
2017-02-01T06:04:56.429Z INFO [ClientCnxn] Opening socket connection to server
...
2017-02-01T06:04:56.429Z INFO [ClientCnxn] Opening socket connection to server
...
2017-02-01T06:04:56.430Z INFO [ClientCnxn] Socket connection established to
..., initiating session
2017-02-01T06:04:56.430Z INFO [ClientCnxn] Socket connection established to
..., initiating session
2017-02-01T06:04:56.435Z INFO [ClientCnxn] Session establishment complete on
server ..., sessionid = 0x759f42ff12406f7, negotiated timeout = 6000
2017-02-01T06:04:56.435Z INFO [ClientCnxn] Session establishment complete on
server ..., sessionid = 0x659a74cf4230f64, negotiated timeout = 6000
2017-02-01T06:04:56.437Z INFO [ZkClient] zookeeper state changed
(SyncConnected)
2017-02-01T06:04:56.436Z INFO [ZkClient] zookeeper state changed
(SyncConnected)
2017-02-01T06:04:56.448Z INFO [ZookeeperConsumerConnector]
[graylog2_...-1485929096393-7617aa40], starting auto committer every 1000 ms
2017-02-01T06:04:56.448Z INFO [ZookeeperConsumerConnector]
[graylog2_...-1485929096393-dfeecc8f], starting auto committer every 1000 ms
2017-02-01T06:04:56.452Z INFO [ZkClient] Waiting for keeper state SyncConnected
2017-02-01T06:04:56.458Z INFO [ClientCnxn] Opening socket connection to server
...
2017-02-01T06:04:56.459Z INFO [ClientCnxn] Socket connection established to
..., initiating session
2017-02-01T06:04:56.462Z INFO [ClientCnxn] Session establishment complete on
server ..., sessionid = 0x159f6bbbd725ac9, negotiated timeout = 6000
2017-02-01T06:04:56.464Z INFO [ZkClient] zookeeper state changed
(SyncConnected)
2017-02-01T06:04:56.465Z INFO [ZookeeperConsumerConnector]
[graylog2_...-1485929096393-25d999de], starting auto committer every 1000 ms
