I'd be grateful if anyone could help point me in the right direction for this... Here's the issue:
I've got a 3-node setup sitting behind an F5 as a POC... After about 5-10 minutes of all three nodes up and running, I start to see occasional "blips" in the web UI and the following entries in the logs of all three systems: 2016-05-12T10:25:24.514-07:00 WARN [ProxiedResource] Node <3116ac6b-604f-4436-955c-1458cb489415> not found while trying to call org.graylog2.rest.resources.system.jobs.RemoteSystemJobResource on it. 2016-05-12T10:25:27.603-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:25:30.910-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:25:37.556-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:25:40.808-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:25:44.663-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:25:47.922-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:25:57.609-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:00.731-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:04.334-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:07.561-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:10.761-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:26.575-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:30.348-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:33.645-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:37.474-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:40.659-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:43.920-07:00 WARN [NodePingThread] Did not find meta info of this node. Re-registering. 2016-05-12T10:26:50.739-07:00 WARN [ProxiedResource] Node <3116ac6b-604f-4436-955c-1458cb489415> not found while trying to call org.graylog2.shared.rest.resources.system.RemoteMetricsResource on it. I did some googling, and the only thing I'm finding is that it's possible the nodes are out of sync time wise.. I checked the system times, and those all look correct: root@gray00 /data]# ntpdate -q ntp0 server 10.201.136.38, stratum 3, offset -0.003738, delay 0.02574 12 May 10:31:40 ntpdate[30786]: adjust time server 10.201.136.38 offset -0.003738 sec [root@gray01 graylog]# ntpdate -q ntp0 server 10.201.136.38, stratum 3, offset -0.000990, delay 0.02576 12 May 10:31:41 ntpdate[28966]: adjust time server 10.201.136.38 offset -0.000990 sec [root@gray02 /data]# ntpdate -q ntp0 server 10.201.136.38, stratum 3, offset 0.001198, delay 0.02582 12 May 10:31:41 ntpdate[30441]: adjust time server 10.201.136.38 offset 0.001198 sec Anyone have any ideas? Here's my setup: 3 nodes: gray00, gray01, gray02 - each node runs one instance of elastic search, standard ports. mongodb, standard ports graylog-server - elastic search ports are on 9350. F5 provides a frontend as "graylog.somewhere.com" ports 12900 & 9000 load balanced to each of the 3 nodes. Sticky sessions enabled on both... so what am I missing here? Graylog configuration settings (same for all 3 nodes, just the listen IPs are different and gray01/gray02 'is_master = false') s_master = true node_id_file = /etc/graylog/server/node-id password_secret = WQBdx6xgWTTykN9LHJhEGxfiSJbeYdaZhHhKEwbvAKQEWkVrl8lgTLvDDkfUtwhe7jgdFDFCBqpmVvY4aea1GyrbQ791UOCv root_password_sha2 = e3ed009797ada49a3fd38a04069b13d5a7f62001a153ed4d9a3da22fa7a75c7b plugin_dir = /usr/share/graylog-server/plugin rest_listen_uri = http://10.201.137.208:12900/ rest_transport_uri = http://graylog.somewhere.com:12900/ rest_enable_gzip = true web_listen_uri = http://10.201.137.208:9000/ web_enable_gzip = true rotation_strategy = count elasticsearch_max_docs_per_index = 20000000 elasticsearch_max_number_of_indices = 20 retention_strategy = delete elasticsearch_shards = 4 elasticsearch_replicas = 1 elasticsearch_index_prefix = graylog allow_leading_wildcard_searches = false allow_highlighting = false elasticsearch_cluster_name = graylog elasticsearch_node_name_prefix = graylog- elasticsearch_discovery_zen_ping_unicast_hosts = gray00:9300, gray01:9300, gray02:9300, gray00:9350, gray01:9350, gray02:9350 elasticsearch_transport_tcp_port = 9350 elasticsearch_discovery_zen_ping_multicast_enabled = false elasticsearch_network_host = gray00 elasticsearch_network_bind_host = gray00 elasticsearch_network_publish_host = gray00 elasticsearch_analyzer = standard output_batch_size = 500 output_flush_interval = 1 output_fault_count_threshold = 5 output_fault_penalty_seconds = 30 processbuffer_processors = 5 outputbuffer_processors = 3 processor_wait_strategy = blocking ring_size = 65536 inputbuffer_ring_size = 65536 inputbuffer_processors = 2 inputbuffer_wait_strategy = blocking message_journal_enabled = true message_journal_dir = /data/graylog/journal lb_recognition_period_seconds = 3 mongodb_uri = mongodb://gray00:27017,gray01:27017,gray02:2701/graylog mongodb_max_connections = 1000 mongodb_threads_allowed_to_block_multiplier = 5 transport_email_enabled = true transport_email_hostname = smtp transport_email_port =25 transport_email_subject_prefix = [graylog] transport_email_from_email = graylog-...@somewhere.com transport_email_web_interface_url = https://graylog.somewhere.com content_packs_dir = /usr/share/graylog-server/contentpacks content_packs_auto_load = grok-patterns.json -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/6c2d6c23-1658-4c09-a179-06dc6273b6b4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
