I am testing out graylog 2.0.2 with elasticsearch 2.3.3 (with 2 nodes for
now for elasticsearch, will become 3 to avoid split-brain ) and mongodb
2.4.14 (in 3-way replica set running on 2 graylog nodes and 1 mongo arbiter
on the loadbalancer in front of graylog ).
However, I'm constantly encountering an error where graylog-server is
unable to connect to Elasticsearch and now I'm not sure why it is not
working. I had tested with the 2.0-Beta before and that worked without
issues.
Details :
-------------------------------------------------------------------------------------------
Elasticsearch -
rpm version: elasticsearch-2.3.3-1.noarch
config:
cluster.name: graylognew
node.name: graylog-es01
path.data: /elasticsearch
network.host: 10.30.20.58
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.30.20.58:9300","10.30.20.59:9300"]
log:
[2016-06-21 09:33:08,599][WARN ][bootstrap ] unable to
install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled
into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2016-06-21 09:33:08,697][INFO ][node ] [graylog-es01]
version[2.3.3], pid[3850], build[218bdf1/2016-05-17T15:40:04Z]
[2016-06-21 09:33:08,697][INFO ][node ] [graylog-es01]
initializing ...
[2016-06-21 09:33:09,034][INFO ][plugins ] [graylog-es01]
modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-06-21 09:33:09,048][INFO ][env ] [graylog-es01]
using [1] data paths, mounts [[/elasticsearch (/dev/md2)]], net
usable_space [733.1gb], net total_space [733.2gb], spins? [possibly], types
[ext4]
[2016-06-21 09:33:09,048][INFO ][env ] [graylog-es01]
heap size [31.8gb], compressed ordinary object pointers [false]
[2016-06-21 09:33:09,048][WARN ][env ] [graylog-es01]
max file descriptors [65535] for elasticsearch process likely too low,
consider increasing to at least [65536]
[2016-06-21 09:33:09,999][INFO ][node ] [graylog-es01]
initialized
[2016-06-21 09:33:09,999][INFO ][node ] [graylog-es01]
starting ...
[2016-06-21 09:33:10,141][INFO ][transport ] [graylog-es01]
publish_address {10.30.20.58:9300}, bound_addresses {10.30.20.58:9300}
[2016-06-21 09:33:10,144][INFO ][discovery ] [graylog-es01]
graylognew/aFMNHpUWScWRtr6AmpMa0Q
[2016-06-21 09:33:13,193][INFO ][cluster.service ] [graylog-es01]
new_master
{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300},
reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-06-21 09:33:13,208][INFO ][http ] [graylog-es01]
publish_address {10.30.20.58:9200}, bound_addresses {10.30.20.58:9200}
[2016-06-21 09:33:13,208][INFO ][node ] [graylog-es01]
started
[2016-06-21 09:33:13,223][INFO ][gateway ] [graylog-es01]
recovered [0] indices into cluster_state
[2016-06-21 09:34:25,603][INFO ][cluster.service ] [graylog-es01]
added
{{graylog-es02}{2Ty5iLUTSbWe5QznunoHkA}{10.30.20.59}{10.30.20.59:9300},},
reason: zen-disco-join(join from
node[{graylog-es02}{2Ty5iLUTSbWe5QznunoHkA}{10.30.20.59}{10.30.20.59:9300}])
<-- nothing else seen after this, was expecting to see graylog2 server
connect to elasticsearch
{
"cluster_name" : "graylognew",
"nodes" : {
"2Ty5iLUTSbWe5QznunoHkA" : {
"name" : "graylog-es02",
"transport_address" : "10.30.20.59:9300",
"host" : "10.30.20.59",
"ip" : "10.30.20.59",
"version" : "2.3.3",
"build" : "218bdf1",
"http_address" : "10.30.20.59:9200",
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 3267,
"mlockall" : false
}
},
"aFMNHpUWScWRtr6AmpMa0Q" : {
"name" : "graylog-es01",
"transport_address" : "10.30.20.58:9300",
"host" : "10.30.20.58",
"ip" : "10.30.20.58",
"version" : "2.3.3",
"build" : "218bdf1",
"http_address" : "10.30.20.58:9200",
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 3850,
"mlockall" : false
}
}
}
}
graylog-es02:/var/log/elasticsearch# curl
http://10.30.20.58:9200/_cluster/health?pretty
{
"cluster_name" : "graylognew",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
-------------------------------------------------------------------------------------------
Graylog
rpmversion : graylog-server-2.0.2-1.noarch
config :
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = .......
root_password_sha2 = ........
root_email = "r...@xyz.com"
root_timezone = US/Eastern
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://graylog-web01:12900/
web_listen_uri = http://10.30.20.60:9000/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 50
retention_strategy = delete
elasticsearch_shards = 8
elasticsearch_replicas = 1
elasticsearch_index_prefix = graylognew
allow_leading_wildcard_searches = false
allow_highlighting = true
elasticsearch_cluster_name = graylognew
elasticsearch_node_name_prefix = graylog2-web01-
elasticsearch_discovery_zen_ping_multicast_enabled = false
elasticsearch_discovery_zen_ping_unicast_hosts = 10.30.20.58:9300,
10.30.20.59:9300
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri =
mongodb://mongouser:password@graylog-web01,graylog-web02/graylog2
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
transport_email_enabled = true
transport_email_hostname = mail.XYZ.com
transport_email_use_auth = false
transport_email_use_tls = false
transport_email_use_ssl = false
transport_email_subject_prefix = [graylog2]
transport_email_from_email = grayl...@graylog-web01.xyz.com
transport_email_web_interface_url = http://graylog-web01:9000
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
log :
2016-06-21T09:36:45.014-04:00 INFO [CmdLineTool] Loaded plugin: Collector
1.0.2 [org.graylog.plugins.collector.CollectorPlugin]
2016-06-21T09:36:45.016-04:00 INFO [CmdLineTool] Loaded plugin: Enterprise
Integration Plugin 1.0.2
[org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2016-06-21T09:36:45.016-04:00 INFO [CmdLineTool] Loaded plugin:
MapWidgetPlugin 1.0.2 [org.graylog.plugins.map.MapWidgetPlugin]
2016-06-21T09:36:45.016-04:00 INFO [CmdLineTool] Loaded plugin: Pipeline
Processor Plugin 1.0.0-beta.4
[org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2016-06-21T09:36:45.017-04:00 INFO [CmdLineTool] Loaded plugin: Anonymous
Usage Statistics 2.0.2
[org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2016-06-21T09:36:45.169-04:00 INFO [CmdLineTool] Running with JVM
arguments: -Xms4g -Xmx4g -XX:NewRatio=1 -XX:+ResizeTLAB
-XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled
-XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC
-XX:-OmitStackTraceInFastThrow
-Djava.library.path=/usr/share/graylog-server/lib/sigar
-Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml
-Dgraylog2.installation_source=rpm
2016-06-21T09:36:48.826-04:00 INFO [InputBufferImpl] Message journal is
enabled.
2016-06-21T09:36:49.243-04:00 INFO [LogManager] Loading logs.
2016-06-21T09:36:49.525-04:00 INFO [LogManager] Logs loading complete.
2016-06-21T09:36:49.526-04:00 INFO [KafkaJournal] Initialized Kafka based
journal at /var/lib/graylog-server/journal
2016-06-21T09:36:49.566-04:00 INFO [InputBufferImpl] Initialized
InputBufferImpl with ring size <65536> and wait strategy
<BlockingWaitStrategy>, running 2 parallel message handlers.
2016-06-21T09:36:49.643-04:00 INFO [cluster] Cluster created with settings
{hosts=[graylog-web01:27017, graylog-web02:27017], mode=MULTIPLE,
requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms',
maxWaitQueueSize=5000}
2016-06-21T09:36:49.644-04:00 INFO [cluster] Adding discovered server
graylog-web01:27017 to client view of cluster
2016-06-21T09:36:49.674-04:00 INFO [cluster] Adding discovered server
graylog-web02:27017 to client view of cluster
2016-06-21T09:36:49.731-04:00 INFO [cluster] No server chosen by
ReadPreferenceServerSelector{readPreference=primary} from cluster
description ClusterDescription{type=UNKNOWN, connectionMode=MULTIPLE,
all=[ServerDescription{address=graylog-web01:27017, type=UNKNOWN,
state=CONNECTING}, ServerDescription{address=graylog-web02:27017,
type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2016-06-21T09:36:49.817-04:00 INFO [connection] Opened connection
[connectionId{localValue:1, serverValue:27913}] to graylog-web02:27017
2016-06-21T09:36:49.817-04:00 INFO [connection] Opened connection
[connectionId{localValue:2, serverValue:27987}] to graylog-web01:27017
2016-06-21T09:36:49.834-04:00 INFO [cluster] Monitor thread successfully
connected to server with description
ServerDescription{address=graylog-web02:27017, type=REPLICA_SET_SECONDARY,
state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 4, 14]},
minWireVersion=0, maxWireVersion=0, maxDocumentSize=16777216,
roundTripTimeNanos=2315552, setName='graylog2Repl',
canonicalAddress=graylog-web02.:27017, hosts=[graylog-web02.:27017,
graylog-web01.:27017], passives=[], arbiters=[graylog.:27017],
primary='graylog-web01.:27017', tagSet=TagSet{[]}, electionId=null,
setVersion=null}
2016-06-21T09:36:49.834-04:00 INFO [cluster] Monitor thread successfully
connected to server with description
ServerDescription{address=graylog-web01:27017, type=REPLICA_SET_PRIMARY,
state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 4, 14]},
minWireVersion=0, maxWireVersion=0, maxDocumentSize=16777216,
roundTripTimeNanos=1859299, setName='graylog2Repl',
canonicalAddress=graylog-web01.:27017, hosts=[graylog-web02.:27017,
graylog-web01.:27017], passives=[], arbiters=[graylog.:27017],
primary='graylog-web01.:27017', tagSet=TagSet{[]}, electionId=null,
setVersion=null}
2016-06-21T09:36:49.840-04:00 INFO [cluster] Discovered cluster type of
REPLICA_SET
2016-06-21T09:36:49.842-04:00 INFO [cluster] Adding discovered server
graylog-web02.:27017 to client view of cluster
2016-06-21T09:36:49.845-04:00 INFO [cluster] Adding discovered server
graylog-web01.:27017 to client view of cluster
2016-06-21T09:36:49.846-04:00 INFO [cluster] Adding discovered server
graylog.:27017 to client view of cluster
2016-06-21T09:36:49.846-04:00 INFO [cluster] Canonical address
graylog-web02.:27017 does not match server address. Removing
graylog-web02:27017 from client view of cluster
2016-06-21T09:36:49.854-04:00 INFO [cluster] Server graylog-web01:27017 is
no longer a member of the replica set. Removing from client view of
cluster.
2016-06-21T09:36:49.855-04:00 INFO [connection] Opened connection
[connectionId{localValue:5, serverValue:27856}] to graylog.:27017
2016-06-21T09:36:49.856-04:00 INFO [cluster] Canonical address
graylog-web01.:27017 does not match server address. Removing
graylog-web01:27017 from client view of cluster
2016-06-21T09:36:49.857-04:00 INFO [connection] Opened connection
[connectionId{localValue:3, serverValue:27914}] to graylog-web02.:27017
2016-06-21T09:36:49.858-04:00 INFO [cluster] Monitor thread successfully
connected to server with description
ServerDescription{address=graylog.:27017, type=REPLICA_SET_ARBITER,
state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 4, 14]},
minWireVersion=0, maxWireVersion=0, maxDocumentSize=16777216,
roundTripTimeNanos=1921928, setName='graylog2Repl',
canonicalAddress=graylog.:27017, hosts=[graylog-web02.:27017,
graylog-web01.:27017], passives=[], arbiters=[graylog.:27017],
primary='graylog-web01.:27017', tagSet=TagSet{[]}, electionId=null,
setVersion=null}
2016-06-21T09:36:49.860-04:00 INFO [cluster] Monitor thread successfully
connected to server with description
ServerDescription{address=graylog-web02.:27017, type=REPLICA_SET_SECONDARY,
state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 4, 14]},
minWireVersion=0, maxWireVersion=0, maxDocumentSize=16777216,
roundTripTimeNanos=1739874, setName='graylog2Repl',
canonicalAddress=graylog-web02.:27017, hosts=[graylog-web02.:27017,
graylog-web01.:27017], passives=[], arbiters=[graylog.:27017],
primary='graylog-web01.:27017', tagSet=TagSet{[]}, electionId=null,
setVersion=null}
2016-06-21T09:36:49.896-04:00 INFO [connection] Opened connection
[connectionId{localValue:4, serverValue:27988}] to graylog-web01.:27017
2016-06-21T09:36:49.899-04:00 INFO [cluster] Monitor thread successfully
connected to server with description
ServerDescription{address=graylog-web01.:27017, type=REPLICA_SET_PRIMARY,
state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 4, 14]},
minWireVersion=0, maxWireVersion=0, maxDocumentSize=16777216,
roundTripTimeNanos=1006498, setName='graylog2Repl',
canonicalAddress=graylog-web01.:27017, hosts=[graylog-web02.:27017,
graylog-web01.:27017], passives=[], arbiters=[graylog.:27017],
primary='graylog-web01.:27017', tagSet=TagSet{[]}, electionId=null,
setVersion=null}
2016-06-21T09:36:49.899-04:00 INFO [cluster] Discovered replica set
primary graylog-web01.:27017
2016-06-21T09:36:49.909-04:00 INFO [connection] Opened connection
[connectionId{localValue:6, serverValue:27989}] to graylog-web01.:27017
2016-06-21T09:36:50.292-04:00 INFO [NodeId] Node ID:
90a4086e-d119-483f-953e-4f34f9524578
2016-06-21T09:36:50.397-04:00 INFO [node]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] version[2.3.2],
pid[29127], build[b9e4a6a/2016-04-21T16:03:47Z]
2016-06-21T09:36:50.398-04:00 INFO [node]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] initializing ...
2016-06-21T09:36:50.408-04:00 INFO [plugins]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] modules [], plugins
[graylog-monitor], sites []
2016-06-21T09:36:53.199-04:00 INFO [node]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] initialized
2016-06-21T09:36:53.309-04:00 INFO [Version] HV000001: Hibernate Validator
5.2.4.Final
2016-06-21T09:36:53.536-04:00 INFO [ProcessBuffer] Initialized
ProcessBuffer with ring size <65536> and wait strategy
<BlockingWaitStrategy>.
2016-06-21T09:36:56.162-04:00 INFO [RulesEngineProvider] No static rules
file loaded.
2016-06-21T09:36:56.386-04:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-21T09:36:56.397-04:00 INFO [OutputBuffer] Initialized OutputBuffer
with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2016-06-21T09:36:56.469-04:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-21T09:36:56.534-04:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-21T09:36:56.637-04:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-21T09:36:56.715-04:00 WARN [GeoIpResolverEngine] GeoIP database
file does not exist: /tmp/GeoLite2-City.mmdb
2016-06-21T09:36:57.591-04:00 INFO [connection] Opened connection
[connectionId{localValue:7, serverValue:27991}] to graylog-web01.:27017
2016-06-21T09:36:58.238-04:00 INFO [ServerBootstrap] Graylog server 2.0.2
(4da1379) starting up
2016-06-21T09:36:58.239-04:00 INFO [ServerBootstrap] JRE: Oracle
Corporation 1.8.0_91 on Linux 3.10.36-el6.ia32e.limeprod.0
2016-06-21T09:36:58.239-04:00 INFO [ServerBootstrap] Deployment: rpm
2016-06-21T09:36:58.239-04:00 INFO [ServerBootstrap] OS: Red Hat
Enterprise Linux Server release 6.8 (Santiago)
2016-06-21T09:36:58.239-04:00 INFO [ServerBootstrap] Arch: amd64
2016-06-21T09:36:58.245-04:00 WARN [DeadEventLoggingListener] Received
unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from
event bus <AsyncEventBus{graylog-eventbus}>
2016-06-21T09:36:58.273-04:00 INFO [PeriodicalsService] Starting 24
periodicals ...
2016-06-21T09:36:58.274-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling
every [1s].
2016-06-21T09:36:58.280-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling
every [60s].
2016-06-21T09:36:58.282-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical
in [0s], polling every [1s].
2016-06-21T09:36:58.285-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s],
polling every [20s].
2016-06-21T09:36:58.598-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running
forever.
2016-06-21T09:36:58.592-04:00 INFO [node]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] starting ...
2016-06-21T09:36:58.600-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.GarbageCollectionWarningThread] periodical,
running forever.
2016-06-21T09:36:58.603-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s],
polling every [30s].
2016-06-21T09:36:58.611-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling
every [300s].
2016-06-21T09:36:58.617-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling
every [10s].
2016-06-21T09:36:58.617-04:00 INFO [IndexRetentionThread] Elasticsearch
cluster not available, skipping index retention checks.
2016-06-21T09:36:58.623-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.NodePingThread] periodical in [0s], polling every
[1s].
2016-06-21T09:36:58.280-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling
every [60s].
2016-06-21T09:36:58.282-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical
in [0s], polling every [1s].
2016-06-21T09:36:58.285-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s],
polling every [20s].
2016-06-21T09:36:58.598-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running
forever.
2016-06-21T09:36:58.592-04:00 INFO [node]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] starting ...
2016-06-21T09:36:58.600-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.GarbageCollectionWarningThread] periodical,
running forever.
2016-06-21T09:36:58.603-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s],
polling every [30s].
2016-06-21T09:36:58.611-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling
every [300s].
2016-06-21T09:36:58.617-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling
every [10s].
2016-06-21T09:36:58.617-04:00 INFO [IndexRetentionThread] Elasticsearch
cluster not available, skipping index retention checks.
2016-06-21T09:36:58.623-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.NodePingThread] periodical in [0s], polling every
[1s].
2016-06-21T09:36:58.628-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling
every [1800s].
2016-06-21T09:36:58.628-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s],
polling every [1s].
2016-06-21T09:36:58.629-04:00 INFO [Periodicals] Starting
[org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling
every [1s].
2016-06-21T09:36:58.630-04:00 INFO [Periodicals] Starting
[org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s],
polling every [300s].
2016-06-21T09:36:58.631-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running
forever.
2016-06-21T09:36:58.631-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical,
running forever.
2016-06-21T09:36:58.632-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s],
polling every [3600s].
2016-06-21T09:36:58.633-04:00 INFO [connection] Opened connection
[connectionId{localValue:8, serverValue:27992}] to graylog-web01.:27017
2016-06-21T09:36:58.640-04:00 INFO [connection] Opened connection
[connectionId{localValue:9, serverValue:27993}] to graylog-web01.:27017
2016-06-21T09:36:58.641-04:00 INFO [connection] Opened connection
[connectionId{localValue:10, serverValue:27994}] to graylog-web01.:27017
2016-06-21T09:36:58.652-04:00 INFO [connection] Opened connection
[connectionId{localValue:11, serverValue:27995}] to graylog-web01.:27017
2016-06-21T09:36:58.659-04:00 INFO [IndexerClusterCheckerThread] Indexer
not fully initialized yet. Skipping periodic cluster check.
2016-06-21T09:36:58.706-04:00 INFO [PeriodicalsService] Not starting
[org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not
configured to run on this node.
2016-06-21T09:36:58.707-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical,
running forever.
2016-06-21T09:36:58.708-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.ConfigurationManagementPeriodical] periodical,
running forever.
2016-06-21T09:36:58.713-04:00 INFO [Periodicals] Starting
[org.graylog2.periodical.LdapGroupMappingMigration] periodical, running
forever.
2016-06-21T09:36:58.715-04:00 INFO [Periodicals] Starting
[org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical
in [300s], polling every [21600s].
2016-06-21T09:36:58.716-04:00 INFO [Periodicals] Starting
[org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical]
periodical in [300s], polling every [21600s].
2016-06-21T09:36:58.717-04:00 INFO [Periodicals] Starting
[org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread]
periodical in [0s], polling every [3600s].
2016-06-21T09:36:58.859-04:00 INFO [transport]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] publish_address
{127.0.0.1:9350}, bound_addresses {127.0.0.1:9350}
2016-06-21T09:36:58.870-04:00 INFO [discovery]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578]
graylognew/YH3f_DKNRGuzMVmX87B3DQ
2016-06-21T09:36:59.121-04:00 INFO [AbstractJerseyService] Enabling CORS
for HTTP endpoint
2016-06-21T09:37:01.875-04:00 WARN [discovery]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] waited for 3s and no
initial state was set by the discovery
2016-06-21T09:37:01.875-04:00 INFO [node]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] started
2016-06-21T09:37:01.917-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:03.891-04:00 INFO [NetworkListener] Started listener
bound to [10.30.20.60:9000]
2016-06-21T09:37:03.894-04:00 INFO [HttpServer] [HttpServer] Started.
2016-06-21T09:37:03.896-04:00 INFO [WebInterfaceService] Started Web
Interface at <http://10.30.20.60:9000/>
2016-06-21T09:37:04.934-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:06.884-04:00 WARN [IndexerSetupService] Could not connect
to Elasticsearch
2016-06-21T09:37:06.884-04:00 INFO [IndexerSetupService] If you're using
multicast, check that it is working in your network and that Elasticsearch
is accessible. Also check that the cluster name setting is correct.
2016-06-21T09:37:06.885-04:00 INFO [IndexerSetupService] See
http://docs.graylog.org/en/2.0/pages/configuring_es.html for details.
2016-06-21T09:37:07.958-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:10.857-04:00 INFO [NetworkListener] Started listener
bound to [graylog-web01:12900]
2016-06-21T09:37:10.858-04:00 INFO [HttpServer] [HttpServer-1] Started.
2016-06-21T09:37:10.860-04:00 INFO [RestApiService] Started REST API at
<http://graylog-web01:12900/>
2016-06-21T09:37:10.864-04:00 INFO [ServiceManagerListener] Services are
healthy
2016-06-21T09:37:10.866-04:00 INFO [InputSetupService] Triggering
launching persisted inputs, node transitioned from Uninitialized [LB:DEAD]
to Running [LB:ALIVE]
2016-06-21T09:37:10.868-04:00 INFO [ServerBootstrap] Services started,
startup times in ms: {MetricsReporterService [RUNNING]=4,
BufferSynchronizerService [RUNNING]=4, InputSetupService [RUNNING]=6,
JournalReader [RUNNING]=7, KafkaJournal [RUNNING]=11, OutputSetupService
[RUNNING]=310, PeriodicalsService [RUNNING]=448, WebInterfaceService
[RUNNING]=5612, IndexerSetupService [RUNNING]=8606, RestApiService
[RUNNING]=12590}
2016-06-21T09:37:10.875-04:00 INFO [ServerBootstrap] Graylog server up and
running.
2016-06-21T09:37:10.975-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:13.632-04:00 INFO [IndexRangesCleanupPeriodical] Skipping
index range cleanup because the Elasticsearch cluster is unreachable or
unhealthy
2016-06-21T09:37:13.989-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:17.000-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:20.010-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:23.020-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:26.030-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:29.046-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
2016-06-21T09:37:32.056-04:00 INFO [zen]
[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578] failed to send join
request to master
[{graylog-es01}{aFMNHpUWScWRtr6AmpMa0Q}{10.30.20.58}{10.30.20.58:9300}],
reason
[RemoteTransportException[[graylog-es01][10.30.20.58:9300][internal:discovery/zen/join]];
nested:
ConnectTransportException[[graylog2-web01-90a4086e-d119-483f-953e-4f34f9524578][127.0.0.1:9350]
connect_timeout[30s]]; nested:
NotSerializableExceptionWrapper[connect_exception: Connection refused:
/127.0.0.1:9350]; ]
...
Why is graylog-server unable to connect to elasticsearch? I can connect to
elasticsearch from the graylog servers without issue via netcat and curl.
There's no firewalls blocking anything and everything is in the same vlan.
Thanks,
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/70d5a81a-ba6b-420c-8b04-0b7489f3a831%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
