BryanDavis has uploaded a new change for review.
https://gerrit.wikimedia.org/r/298382
Change subject: logstash: Remove all _* fields from gelf records
......................................................................
logstash: Remove all _* fields from gelf records
Fields named _id, _index and _type were retained when GELF filtering was
initially setup in dfb0a19. These field names are significant in
Elasticsearch, but they are added by the Elasticsearch server itself as
metadata for each log event passed in by Logstash. They should not be
passed in via the application. Doing so results in mapping conflicts
that discard the entire event such as "MapperParsingException[failed to
parse]; nested: IllegalStateException[Mixing up field types: class
org.elasticsearch.index.mapper.core.StringFieldMapper$StringFieldType !=
class org.elasticsearch.index.mapper.internal.IdFieldMapper$IdFieldType
on field _id];".
Change-Id: I6306f76052c163e1edbf044343d11514b5b20803
---
M files/logstash/filter-gelf.conf
1 file changed, 3 insertions(+), 3 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/82/298382/1
diff --git a/files/logstash/filter-gelf.conf b/files/logstash/filter-gelf.conf
index ed39bb4..1ce9164 100644
--- a/files/logstash/filter-gelf.conf
+++ b/files/logstash/filter-gelf.conf
@@ -84,10 +84,10 @@
remove_field => [ "full_message" ]
}
- # Get rid of a whole bunch of useless _* fields that are duplicated.
- # We must keep _id, _index, and _type
+ # Get rid of a whole bunch of useless _* fields that are duplicated or
+ # collide with internal Elasticsearch meta data.
prune {
- blacklist_names => [ "^_(?!id$|index$|type$).*" ]
+ blacklist_names => [ "^_.*" ]
}
} # end [type] == "gelf"
--
To view, visit https://gerrit.wikimedia.org/r/298382
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6306f76052c163e1edbf044343d11514b5b20803
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: BryanDavis <bda...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
