[graylog2] Re: Using the Beats inputs - forwarder configuration
Hi Jochen, I am using Filebeat 1.3.1: sudo rpm -qa | grep filebeat filebeat-1.3.1-1.x86_64 What I did just notice as I looked at the post from @cr0c that I have made a total school boy error. As i checked that the logs had been written to I noticed that the logs in the filebeat yaml did not exist. I had lifted the yaml from an Ubuntu estate and the Centos log names are different. I switched the yaml to monitor /var/log/secure and /var/log/messages restarted the filebeat service and surprise surprise data started flowing into Graylog. I am interested in investigating the Graylog Collector Sidecar and I notice that Fluentd is a configurable forwarder which is an option I should also look at. My main goal is to limit user access to specific data by LDAP groups and the forwarders will be running on syslog servers located in AWS accounts that are specific to projects/environments. Ansible/Packer and Terraform will be used to spin up environments so I need to test each available configuration combination. Thanks, Chris. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/7ecc7cc3-faa8-4c8f-8a89-4bcb92bb4a11%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Using the Beats inputs - forwarder configuration
Hi Jochen, I set up a filebeat forwarder on the Graylog server just to test localhost forwarding with the following configurations but even though Graylogs inputs page shows an active connection no data is being ingested: - bind_address: 0.0.0.0 - override_source: admin - port: 5044 - recv_buffer_size: 1048576 - tcp_keepalive: false - tls_cert_file: ** - tls_client_auth: disabled - tls_client_auth_cert_file: ** - tls_enable: false - tls_key_file: ** - tls_key_password: Filebeat (/etc/filebeat/filebeat.yml): filebeat: prospectors: - paths: - /var/log/auth.log - /var/log/syslog input_type: log document_type: syslog registry_file: /var/lib/filebeat/registry output: logstash: hosts: ["127.0.0.1:5044"] bulk_max_size: 1024 logging: files: rotateeverybytes: 10485760 # = 10MB I took out the TLS configuration just to rule that out as a problem and the filebeat.yml is pretty basic. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/84fb33a7-0fd8-491d-9a83-208a4c321350%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Using the Beats inputs - forwarder configuration
I am really interested in using the Beats forwarders from Elastic to send logs to Graylog as I have used Beats in ELK stacks previously. I am curious as to what the beats forwarders configuration should be. I am used to using Logstash to parse logs before sending to Elasticsearch so I am wondering how the Graylog Beats input works? Is the data sent directly to Elasticsearch so the forwarder output should be Elasticsearch. I ask this as I noticed a comment on the plugin from Joschi saying that Logstash is the correct output. If this is a working forwarder for an ELK stack what needs to change for Graylog? filebeat: prospectors: - paths: - /var/log/auth.log - /var/log/syslog # - /var/log/*.log input_type: log document_type: syslog registry_file: /var/lib/filebeat/registry output: logstash: hosts: ["172.31.31.4:5044","172.31.23.200:5044"] # configure logstash plugin to loadbalance events between the logstash instances loadbalance: true bulk_max_size: 1024 tls: certificate_authorities: ["/etc/pki/tls/certs/logstash-node2fwd.crt","/etc/pki/tls/certs/logstash-forwarder.crt"] shipper: logging: files: rotateeverybytes: 10485760 # = 10MB -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/4e90a272-4890-41cd-b169-40b7dff69189%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Graylog 2.0.1 Web Interface Issue
Hi Jochen, I couldn't find the 'graylog-ctl' scripts so I wasn't sure where the json files has come from on this server. I have been changing the server.conf manually but I got to the point where I had spent too much time on this server. The environment is due to be destroyed and the newly deployed environment has an Ubuntu packaged version that is working. I am happy that I can stand up a Graylog 2.1.1 from scratch so I will bow out defeated on this one and look at building a new ami using Graylog 2.1.1 and Centos 7. Thank you for all your input, Chris. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/d6b3cdf6-0373-4cb3-83f6-66056f8d3012%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Graylog 2.0.1 Web Interface Issue
Hi Jochen,
No reverse proxies in front of Graylog and I updated the settings to look
like this but the error persists (minus the /api reference):
/etc/graylog/server/server.conf
rest_listen_uri = http://0.0.0.0:12900/
web_listen_uri = http://0.0.0.0:9000/
web_endpoint_uri = http://MY_AWS_RT53_DNS:12900/
/etc/graylog/graylog-settings.json
{
"timezone": "Etc/UTC",
"smtp_server": "",
"smtp_port": 587,
"smtp_user": "",
"smtp_password": "",
"smtp_from_email": null,
"smtp_web_url": null,
"smtp_no_tls": false,
"smtp_no_ssl": false,
"master_node": "10.0.99.166",
"local_connect": false,
"current_address": "10.0.99.166",
"last_address": "10.0.99.166",
"enforce_ssl": false,
"journal_size": 1,
"internal_logging": true,
"custom_attributes": {
}
}
Is the external_rest_uri a valid setting? I am finding more Graylog servers
in the estate as I investigate, they are running different versions on
different Linux OS's and they suffer from the same web interface error.
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/6343b8ff-ceed-4823-ad7d-da0eb988b1db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Graylog 2.1.1 Web Interface Problem
Hi Jochen, You are correct, I was indeed missing the */api/ from the URL. I can now log in and I will start looking at using some beats forwarders to get some data into GrayLog/ElasticSearch. Cheers, Chris. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/0ca00c18-7f99-4b77-80a3-e82192b3ded1%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Graylog 2.1.1 Web Interface Problem
Today, I tried to install graylog 2.1.1 in new Amazon Instance to test
features of new graylog. After I installed elastic search 2.4.0, mongodb
3.2.9 and graylog 2.1.1, I configured elasticsearch.yml and graylog config
as below. Then, even though graylog server is up, running and elastic
search added graylog node in logs, I encountered weird problem. Then I
typed the graylog server ip ( <"my amazon instance public ip">:9000 ) in
chrome and safari. However, when I entered my credentials ( admin/graylog
password) and clicked signin, nothing was fired. Then 15-30 seconds later,
graylog web interface gave an error as below:
We are experiencing problems connecting to the Graylog server running on
*http://172.31.29.124:12900/api/*. Please verify that the server is healthy
and working correctly..
My graylog config looks like this (/etc/graylog/server/server.conf):
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = SECRECT
root_password_sha2 = SECRET
root_timezone = GMT
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://0.0.0.0:12900/api/
external_rest_uri: http://MY_AWS_RT53_DNS/api
web_listen_uri = http://0.0.0.0:9000/
rotation_strategy = count
elasticsearch_max_docs_per_index = 2000
rotation_strategy = count
elasticsearch_max_docs_per_index = 2000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 1
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_cluster_name = graylog
elasticsearch_discovery_zen_ping_unicast_hosts = 172.31.29.124:9300
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32
My elasticsearch yml file looks like this
(/etc/elasticsearch/elasticsearch.yml):
cluster.name: graylog
path.data: /elasticsearch/data/
path.logs: /var/log/elasticsearch/
script.inline: false
script.indexed: false
script.file: false
network.host: 172.31.29.124
discovery.zen.ping.timeout: 10s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["172.31.29.124:9300"]
This is the graylog server log (/var/log/graylog-server/server.log):
2016-09-21T08:45:59.563Z INFO [CmdLineTool] Loaded plugin: Elastic Beats
Input 1.1.1 [org.graylog.plugins.beats.BeatsInputPlugin]
2016-09-21T08:45:59.564Z INFO [CmdLineTool] Loaded plugin: Collector 1.1.1
[org.graylog.plugins.collector.CollectorPlugin]
2016-09-21T08:45:59.565Z INFO [CmdLineTool] Loaded plugin: Enterprise
Integration Plugin 1.1.1
[org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2016-09-21T08:45:59.565Z INFO [CmdLineTool] Loaded plugin: MapWidgetPlugin
1.1.1 [org.graylog.plugins.map.MapWidgetPlugin]
2016-09-21T08:45:59.565Z INFO [CmdLineTool] Loaded plugin: Pipeline
Processor Plugin 1.1.1
[org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2016-09-21T08:45:59.566Z INFO [CmdLineTool] Loaded plugin: Anonymous Usage
Statistics 2.1.1 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2016-09-21T08:45:59.676Z INFO [CmdLineTool] Running with JVM arguments:
-Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC
-XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC
-XX:-OmitStackTraceInFastThrow
-Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml
-Djava.library.path=/usr/share/graylog-server/lib/sigar
-Dgraylog2.installation_source=rpm
2016-09-21T08:46:01.979Z INFO [InputBufferImpl] Message journal is enabled.
2016-09-21T08:46:02.003Z INFO [NodeId] Node ID:
a5e73742-5454-49d7-a089-eb3beb6443b8
2016-09-21T08:46:02.202Z INFO [LogManager] Loading logs.
2016-09-21T08:46:02.257Z INFO [LogManager] Logs loading complete.
2016-09-21T08:46:02.257Z INFO [KafkaJournal] Initialized Kafka based
journal at /var/lib/graylog-server/journal
2016-09-21T08:46:02.274Z INFO [InputBufferImpl] Initialized
InputBufferImpl with ring size <65536> and wait strategy
, running 2 parallel message handlers.
2016-09-21T08:46:02.300Z INFO [cluster] Cluster created with settings
{hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN,
serverSelectionTimeout='3 ms', maxWaitQueueSize=5000}
2016-09-21T08:46:02.355Z INFO
