Thanks for sharing Stephane! Just an fyi, we do also recommend setting the type mapping for strings like you've shown, as indicated here - https://github.com/apache/metron/tree/master/metron-platform/metron-elasticsearch#type-mappings .
On Thu, Apr 25, 2019 at 6:33 AM <[email protected]> wrote: > I realize that I’ve missed a part of the story regarding shards. The good > size for shards is around 40~50GB. So, if your index grows up to 200 or > 300GB, you of course need to increase the number of shards to come back > around this size. > > > > This is also why I’d suggest to have yyyy.MM.dd in the “Elasticsearch Date > Format” configuration to not create some hourly indices that will be very > small. > > > > Stéphane > > > > *From:* Nick Allen [mailto:[email protected]] > *Sent:* Thursday, April 25, 2019 14:25 > *To:* DAVY Stephane OBS/CSO > *Subject:* Re: About Elastic templates > > > > Thanks for sharing Stéphane! > > > > > > > > On Thu, Apr 25, 2019 at 5:58 AM <[email protected]> wrote: > > Hello all, > > > > As we heavily use Elasticsearch in our company, with some support from > Elastic company, I’d like to share with about index and template. Here is > the starting template I use: > > > > { > > "*<template_name>*": { > > "template": "*<sensor>*_index_*", > > "settings": { > > "index": { > > "number_of_shards": "1", > > "number_of_replicas": "1" > > } > > }, > > "mappings": { > > "_default_": { > > "dynamic_templates": [ > > { > > "strings_as_keywords": { > > "match_mapping_type": "string", > > "mapping": { > > "type": "keyword" > > } > > } > > } > > ], > > "properties": { > > "timestamp": { > > "type": "date" > > }, > > "@version": { > > "type": "text" > > }, > > "ip_dst_addr": { > > "type": "ip" > > }, > > "ip_src_addr": { > > "type": "ip" > > }, > > "metron_alert": { > > "type": "nested" > > } > > } > > } > > }, > > "aliases": {} > > } > > } > > > > Of course, replace the red parts with your config. > > > > What we do here is to start with shard = 1, instead of shard = 5, which is > the default, and a really bad default according to Elastic support. If you > have small indices and a lot of shards, you will kill your Elastic > performances. > > Regarding fields type: > > - We set the default to “keyword” as “string” is now deprecated > > - We specialize all the non-string fields with their real type. > The “ip” is really useful, it allows CIDR usage in queries > > - The metron_alert one is needed, otherwise the different GUI > exhibits some errors. > > > > > > I hope this helps > > > > Stéphane > > _________________________________________________________________________________________________________________________ > > > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu > ce message par erreur, veuillez le signaler > > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > > they should not be distributed, used or copied without authorisation. > > If you have received this email in error, please notify the sender and delete > this message and its attachments. > > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > > Thank you. > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu > ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and delete > this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. > >
