I have confirmed that’s the behavior as well. In event.rb:
def init_timestamp(o)
begin
timestamp = o ? LogStash::Timestamp.coerce(o) : LogStash::Timestamp.now
You can see that if the @timestamp field already exists, it uses that for the
timestamp, otherwise uses the current time.
And then in elasticsearch.rb, it formats the YYYY.MM.DD with the timestamp:
index = event.sprintf(@index)
where sprintf uses the @timestamp field.
So the key is to make sure either your client sets @timestamp or you use the
date field on the log stash side.
Thanks!
On September 30, 2014 at 2:55:32 PM, Brian (brian.from...@gmail.com) wrote:
Matt,
Assuming your logstash configurations correctly set the @timestamp field, then
logstash will store the document in the day that is specified by the @timestamp
field.
I have verified this behavior by observation over the time we have been using
the ELK stack.
For example, we have a Perl CGI script that is used to emulate a customer
service. It has a hard-coded ISO-8601 date string which our logstash
configuration finds before it notices the syslog date. And so that log entry
ends up in the day in the past that the hard-coded string specifies. And then
curator cleans it up each and every day.
Bottom line: logstash already respects the day in the @timestamp when storing
data in ES.
Brian
On Tuesday, September 30, 2014 2:31:59 PM UTC-4, Matt Hughes wrote:
I have a logstash-forwarder client sending events to lumberjack ->
elasticsearch to timestamped logstash indices. How does logstash decide what
*day* index to put the document in. Does it look at @timestamp? @timestamp is
just generated when the document is received, correct? So if you logged an
event on a client at 11 pm UTC but it didn't make it to elasticsearch until 1am
UTC the next day, which index would it go in? Would it go in the day it was
created or would it go in the day it got to elasticsearch?
If the latter, is there a way to force logstash to respect a date field in the
original log event?
--
You received this message because you are subscribed to a topic in the Google
Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/xCDVA7zTO6w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3931b0d7-6923-4dce-a524-33b49d04af01%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.542afdad.625558ec.2008%40Matthews-MacBook-Pro.local.
For more options, visit https://groups.google.com/d/optout.
