Just checked on the length issue - we should be good - https://github.com/elastic/elasticsearch/issues/8079
On Fri, Jan 26, 2018 at 3:37 PM, James Sirota <jsir...@apache.org> wrote: > Seems reasonable to me. The only thing is that it may make the index > names too long. Not sure if that matters to ES or not > > 26.01.2018, 15:32, "Simon Elliston Ball" <si...@simonellistonball.com>: > > +1 on this. The idea of a default broad matching template should also > include an order entry to avoid conflicts with more specific templates, and > we should then document the need for a higher order value in all per-source > index templates. > > > > In terms of production migration, I think we may want to provide some > detailed documentation in the upgrade guide on this, because there will be > people with a lot of existing indices that will be difficult to handle. We > may also need some tooling, but I expect docs would do the job. What do > people think about migration? > > > > Simon > > > >> One other benefit of this revised approach - we can more effectively > use > >> index template patterns to specify our base set of Metron property > types. > >> Call me crazy, but I think we should be able to do something like: > >> > >> <metron_default_template> > >> > >> { > >> *"template": "metron_*",* > >> "mappings": { > >> "metron_doc": { > >> "dynamic_templates": [ > >> { > >> "geo_location_point": { > >> "match": "enrichments:geo:*:location_point", > >> "match_mapping_type": "*", > >> "mapping": { > >> "type": "geo_point" > >> } > >> } > >> }, > >> { > >> "geo_country": { > >> "match": "enrichments:geo:*:country", > >> "match_mapping_type": "*", > >> "mapping": { > >> "type": "keyword" > >> } > >> } > >> }, > >> { > >> "geo_city": { > >> "match": "enrichments:geo:*:city", > >> "match_mapping_type": "*", > >> "mapping": { > >> "type": "keyword" > >> } > >> } > >> }, > >> { > >> "geo_location_id": { > >> "match": "enrichments:geo:*:locID", > >> "match_mapping_type": "*", > >> "mapping": { > >> "type": "keyword" > >> } > >> } > >> }, > >> { > >> "geo_dma_code": { > >> "match": "enrichments:geo:*:dmaCode", > >> "match_mapping_type": "*", > >> "mapping": { > >> "type": "keyword" > >> } > >> } > >> }, > >> { > >> "geo_postal_code": { > >> "match": "enrichments:geo:*:postalCode", > >> "match_mapping_type": "*", > >> "mapping": { > >> "type": "keyword" > >> } > >> } > >> }, > >> { > >> "geo_latitude": { > >> "match": "enrichments:geo:*:latitude", > >> "match_mapping_type": "*", > >> "mapping": { > >> "type": "float" > >> } > >> } > >> }, > >> { > >> "geo_longitude": { > >> "match": "enrichments:geo:*:longitude", > >> "match_mapping_type": "*", > >> "mapping": { > >> "type": "float" > >> } > >> } > >> }, > >> { > >> "timestamps": { > >> "match": "*:ts", > >> "match_mapping_type": "*", > >> "mapping": { > >> "type": "date", > >> "format": "epoch_millis" > >> } > >> } > >> }, > >> { > >> "threat_triage_score": { > >> "mapping": { > >> "type": "float" > >> }, > >> "match": "threat:triage:*score", > >> "match_mapping_type": "*" > >> } > >> }, > >> { > >> "threat_triage_reason": { > >> "mapping": { > >> "type": "text", > >> "fielddata": "true" > >> }, > >> "match": "threat:triage:rules:*:reason", > >> "match_mapping_type": "*" > >> } > >> }, > >> { > >> "threat_triage_name": { > >> "mapping": { > >> "type": "text", > >> "fielddata": "true" > >> }, > >> "match": "threat:triage:rules:*:name", > >> "match_mapping_type": "*" > >> } > >> } > >> > >> ]}} > >> > >> That means that for every new sensor we bring on board we can skip > >> adding that boiler plate mapping config to every new template. > >> > >> On Wed, Jan 24, 2018 at 6:34 PM, Michael Miklavcic < > >> michael.miklav...@gmail.com> wrote: > >> > >>> I hear you Ali. I think this type of change would actually ease issues > >>> with downtime because it offers an easy path to migrating existing > indices. > >>> I'd have to review the specifics in the ES docs again, but I believe > you > >>> could duplicate the old indexes and migrate them to "metron_" in > advance of > >>> the upgrade, and then consume new data to the new index pattern/name > after > >>> the upgrade. That should be pretty seamless, I think. I guess it > depends on > >>> how you're using ES. > >>> > >>> On Wed, Jan 24, 2018 at 4:08 PM, Ali Nazemian <alinazem...@gmail.com> > >>> wrote: > >>> > >>>> Hi All, > >>>> > >>>> I just wanted to say it would be great if we can be careful with > these > >>>> type > >>>> of changes. From the development point of view, it is just a few > lines of > >>>> code which can provide multiple advantages, but for live large-scale > >>>> Metron > >>>> platforms, some of these changes might be really expensive to > address with > >>>> zero-downtime. > >>>> > >>>> Cheers, > >>>> Ali > >>>> > >>>> On Thu, Jan 25, 2018 at 9:29 AM, Otto Fowler < > ottobackwa...@gmail.com> > >>>> wrote: > >>>> > >>>>> +1 > >>>>> > >>>>> On January 24, 2018 at 16:28:42, Nick Allen (n...@nickallen.org) > wrote: > >>>>> > >>>>> +1 to a standard prefix for all Metron indices. I've had the same > >>>> thought > >>>>> myself and you laid out the advantages well. > >>>>> > >>>>> On Wed, Jan 24, 2018 at 3:47 PM zeo...@gmail.com <zeo...@gmail.com> > >>>> wrote: > >>>>>> I agree with having a metron_ prefix for ES indexes, and the > timing. > >>>>>> > >>>>>> Jon > >>>>>> > >>>>>> On Wed, Jan 24, 2018 at 3:20 PM Michael Miklavcic < > >>>>>> michael.miklav...@gmail.com> wrote: > >>>>>> > >>>>>>> With the completion of https://github.com/apache/metron/pull/840 > >>>>>>> (METRON-939: Upgrade ElasticSearch and Kibana), we have the > makings > >>>> for > >>>>> a > >>>>>>> major release rev of Metron in the upcoming release (currently > >>>> slotted > >>>>> to > >>>>>>> 0.4.3, I believe). Since there are non-backwards compatible > changes > >>>>>>> pertaining to ES indexing, it seems like a good opportunity to > >>>> revisit > >>>>>> our > >>>>>>> index naming standards. > >>>>>>> > >>>>>>> I propose we add a simple prefix "metron_" to all Metron indexes. > >>>> There > >>>>>> are > >>>>>>> numerous reasons for doing so > >>>>>>> > >>>>>>> - removes the likelihood of index name collisions when we perform > >>>>>>> operations on index wildcard names, e.g. "enrichment_*, > indexing_*, > >>>>>>> etc.". > >>>>>>> - ie, this allows us to be more friendly in a multi-tenant ES > >>>>>>> environment for relatively low engineering cost. > >>>>>>> - simplifies the Kibana dashboard a bit. We currently needed to > >>>>>> create a > >>>>>>> special index pattern in order to accommodate multi-index pattern > >>>>>>> matching > >>>>>>> across all metron-specific indexes. Using metron_* would be much > >>>>>> simpler > >>>>>>> and less prone to error. > >>>>>>> - easier for customers to debug and identify Metron-specific > indexes > >>>>>> and > >>>>>>> associated data > >>>>>>> > >>>>>>> The reason for making these changes now is that we already have > >>>>> breaking > >>>>>>> changes with ES. Leveraging existing indexed data rather than > >>>> deleting > >>>>>>> indexes and starting from scractch already requires a > >>>>>> re-indexing/migration > >>>>>>> step, so there is no additional effort on the part of users if > they > >>>>>> choose > >>>>>>> to attempt a migration. It further makes sense with our current > work > >>>>>>> towards upgrading Solr. > >>>>>>> > >>>>>>> We already have a battery of integration and manual tests after > the > >>>> ES > >>>>>>> upgrade work that can be leveraged to validate the changes. > >>>>>>> > >>>>>>> Mike Miklavcic > >>>>>> > >>>>>> -- > >>>>>> > >>>>>> Jon > >>>> > >>>> -- > >>>> A.Nazemian > > ------------------- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org > >