Re: Question about Logstash Joining ES Cluster and Index
Brian,
I've tried using the http protocol and it did not work. The Cluster sees
the logstash node, but it's not creating the index still.
Thanks,
Gerald
On Friday, November 14, 2014 4:26:22 PM UTC-5, @SQLBigG wrote:
My setup:
Logstash Node parsing NXLog eventlogs from windows servers and sending
them to a 3 node ES Cluster.
Here's my logstash conf file:
input {
tcp {
type = eventlog
host = logstash01
port = 3515
codec = 'json'
}
}
output {
elasticsearch
{
index = logstash-events-np-%{+-MM-dd}
}
}
Here is the elasticsearch.yml file on my logstash server:
cluster.name: ELCluster
node.name: logstash01
discovery.zen.ping.unicasthosts: [elsearch01, elsearch02] (master
nodes)
So when I look at HEAD, I can see that the logstash node has joined the
cluster successfully, but my index is not being built. In the logstash
log, I see the following:
log4j, [2014-11-14T13:22:49.842] WARN: org.elasticsearch.discovery:
[logstash-logstash01-29863-2036] waited for 30s and no initial state was
set by the discovery
Not sure why it's not building my index as I know there is data coming in
from about 50 servers or so. I'm a little lost at this point as I was able
to get this far and don't know if I am missing a setting somewhere. All
servers are on the same network segment and there are no firewalls on the
servers or between the servers. I was able to build basic indexes without
an issue from the ES cluster using the basic commands so I know it has the
ability to create an index.
Originally, the current logstash server and elasticsearch server were our
PoC servers and were successful and now we wanted to build a cluster for ES
since it was going to be handling a large amount of data.
Thanks in advance.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1d852a5d-8621-4b76-b127-c7a886c00561%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Question about Logstash Joining ES Cluster and Index
My setup:
Logstash Node parsing NXLog eventlogs from windows servers and sending them
to a 3 node ES Cluster.
Here's my logstash conf file:
input {
tcp {
type = eventlog
host = logstash01
port = 3515
codec = 'json'
}
}
output {
elasticsearch
{
index = logstash-events-np-%{+-MM-dd}
}
}
Here is the elasticsearch.yml file on my logstash server:
cluster.name: ELCluster
node.name: logstash01
discovery.zen.ping.unicasthosts: [elsearch01, elsearch02] (master nodes)
So when I look at HEAD, I can see that the logstash node has joined the
cluster successfully, but my index is not being built. In the logstash
log, I see the following:
log4j, [2014-11-14T13:22:49.842] WARN: org.elasticsearch.discovery:
[logstash-logstash01-29863-2036] waited for 30s and no initial state was
set by the discovery
Not sure why it's not building my index as I know there is data coming in
from about 50 servers or so. I'm a little lost at this point as I was able
to get this far and don't know if I am missing a setting somewhere. All
servers are on the same network segment and there are no firewalls on the
servers or between the servers. I was able to build basic indexes without
an issue from the ES cluster using the basic commands so I know it has the
ability to create an index.
Originally, the current logstash server and elasticsearch server were our
PoC servers and were successful and now we wanted to build a cluster for ES
since it was going to be handling a large amount of data.
Thanks in advance.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8166101a-4052-4ef0-be30-672aaefaa85d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Question about Logstash Joining ES Cluster and Index
I highly recommend that you use the HTTP output. Works great, is immune to
the ES version, and there are no performance issues that I've seen. It Just
Works.
For example, here's my sample logstash configuration's output settings:
output {
# Uncomment for testing only:
# stdout { codec = rubydebug }
# Elasticsearch
elasticsearch {
# Specify http (with or without quotes around http) to direct the
# output as JSON documents via the Elasticsearch HTTP REST API
protocol = http
codec = json
manage_template = false
# Or whatever target ES host is required
host = localhost
# Or whatever _type is desired:
index_type = sample
}
}
As you can probably surmise, I have my own default index creation template
so there's no need to splatter it all over creation; logstash runs better
on the host on which it's gathering the log files and I vastly prefer one
central index template than keeping a bazillion logstash configurations in
perfect sync. And if we happen replace logstash for something else, then I
still have my index creation templates.
Hope this helps!
Brian
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/27854489-1f4d-4ebd-883c-64dc6235eed4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
