+1 on this. The idea of a default broad matching template should also include
an order entry to avoid conflicts with more specific templates, and we should
then document the need for a higher order value in all per-source index
templates.
In terms of production migration, I think we may want to provide some detailed
documentation in the upgrade guide on this, because there will be people with a
lot of existing indices that will be difficult to handle. We may also need some
tooling, but I expect docs would do the job. What do people think about
migration?
Simon
>
> One other benefit of this revised approach - we can more effectively use
> index template patterns to specify our base set of Metron property types.
> Call me crazy, but I think we should be able to do something like:
>
> <metron_default_template>
>
> {
> *"template": "metron_*",*
> "mappings": {
> "metron_doc": {
> "dynamic_templates": [
> {
> "geo_location_point": {
> "match": "enrichments:geo:*:location_point",
> "match_mapping_type": "*",
> "mapping": {
> "type": "geo_point"
> }
> }
> },
> {
> "geo_country": {
> "match": "enrichments:geo:*:country",
> "match_mapping_type": "*",
> "mapping": {
> "type": "keyword"
> }
> }
> },
> {
> "geo_city": {
> "match": "enrichments:geo:*:city",
> "match_mapping_type": "*",
> "mapping": {
> "type": "keyword"
> }
> }
> },
> {
> "geo_location_id": {
> "match": "enrichments:geo:*:locID",
> "match_mapping_type": "*",
> "mapping": {
> "type": "keyword"
> }
> }
> },
> {
> "geo_dma_code": {
> "match": "enrichments:geo:*:dmaCode",
> "match_mapping_type": "*",
> "mapping": {
> "type": "keyword"
> }
> }
> },
> {
> "geo_postal_code": {
> "match": "enrichments:geo:*:postalCode",
> "match_mapping_type": "*",
> "mapping": {
> "type": "keyword"
> }
> }
> },
> {
> "geo_latitude": {
> "match": "enrichments:geo:*:latitude",
> "match_mapping_type": "*",
> "mapping": {
> "type": "float"
> }
> }
> },
> {
> "geo_longitude": {
> "match": "enrichments:geo:*:longitude",
> "match_mapping_type": "*",
> "mapping": {
> "type": "float"
> }
> }
> },
> {
> "timestamps": {
> "match": "*:ts",
> "match_mapping_type": "*",
> "mapping": {
> "type": "date",
> "format": "epoch_millis"
> }
> }
> },
> {
> "threat_triage_score": {
> "mapping": {
> "type": "float"
> },
> "match": "threat:triage:*score",
> "match_mapping_type": "*"
> }
> },
> {
> "threat_triage_reason": {
> "mapping": {
> "type": "text",
> "fielddata": "true"
> },
> "match": "threat:triage:rules:*:reason",
> "match_mapping_type": "*"
> }
> },
> {
> "threat_triage_name": {
> "mapping": {
> "type": "text",
> "fielddata": "true"
> },
> "match": "threat:triage:rules:*:name",
> "match_mapping_type": "*"
> }
> }
>
> ]}}
>
> That means that for every new sensor we bring on board we can skip
> adding that boiler plate mapping config to every new template.
>
>
>
> On Wed, Jan 24, 2018 at 6:34 PM, Michael Miklavcic <
> [email protected]> wrote:
>
>> I hear you Ali. I think this type of change would actually ease issues
>> with downtime because it offers an easy path to migrating existing indices.
>> I'd have to review the specifics in the ES docs again, but I believe you
>> could duplicate the old indexes and migrate them to "metron_" in advance of
>> the upgrade, and then consume new data to the new index pattern/name after
>> the upgrade. That should be pretty seamless, I think. I guess it depends on
>> how you're using ES.
>>
>> On Wed, Jan 24, 2018 at 4:08 PM, Ali Nazemian <[email protected]>
>> wrote:
>>
>>> Hi All,
>>>
>>> I just wanted to say it would be great if we can be careful with these
>>> type
>>> of changes. From the development point of view, it is just a few lines of
>>> code which can provide multiple advantages, but for live large-scale
>>> Metron
>>> platforms, some of these changes might be really expensive to address with
>>> zero-downtime.
>>>
>>> Cheers,
>>> Ali
>>>
>>> On Thu, Jan 25, 2018 at 9:29 AM, Otto Fowler <[email protected]>
>>> wrote:
>>>
>>>> +1
>>>>
>>>>
>>>> On January 24, 2018 at 16:28:42, Nick Allen ([email protected]) wrote:
>>>>
>>>> +1 to a standard prefix for all Metron indices. I've had the same
>>> thought
>>>> myself and you laid out the advantages well.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Jan 24, 2018 at 3:47 PM [email protected] <[email protected]>
>>> wrote:
>>>>
>>>>> I agree with having a metron_ prefix for ES indexes, and the timing.
>>>>>
>>>>> Jon
>>>>>
>>>>> On Wed, Jan 24, 2018 at 3:20 PM Michael Miklavcic <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> With the completion of https://github.com/apache/metron/pull/840
>>>>>> (METRON-939: Upgrade ElasticSearch and Kibana), we have the makings
>>> for
>>>> a
>>>>>> major release rev of Metron in the upcoming release (currently
>>> slotted
>>>> to
>>>>>> 0.4.3, I believe). Since there are non-backwards compatible changes
>>>>>> pertaining to ES indexing, it seems like a good opportunity to
>>> revisit
>>>>> our
>>>>>> index naming standards.
>>>>>>
>>>>>> I propose we add a simple prefix "metron_" to all Metron indexes.
>>> There
>>>>> are
>>>>>> numerous reasons for doing so
>>>>>>
>>>>>> - removes the likelihood of index name collisions when we perform
>>>>>> operations on index wildcard names, e.g. "enrichment_*, indexing_*,
>>>>>> etc.".
>>>>>> - ie, this allows us to be more friendly in a multi-tenant ES
>>>>>> environment for relatively low engineering cost.
>>>>>> - simplifies the Kibana dashboard a bit. We currently needed to
>>>>> create a
>>>>>> special index pattern in order to accommodate multi-index pattern
>>>>>> matching
>>>>>> across all metron-specific indexes. Using metron_* would be much
>>>>> simpler
>>>>>> and less prone to error.
>>>>>> - easier for customers to debug and identify Metron-specific indexes
>>>>> and
>>>>>> associated data
>>>>>>
>>>>>>
>>>>>> The reason for making these changes now is that we already have
>>>> breaking
>>>>>> changes with ES. Leveraging existing indexed data rather than
>>> deleting
>>>>>> indexes and starting from scractch already requires a
>>>>> re-indexing/migration
>>>>>> step, so there is no additional effort on the part of users if they
>>>>> choose
>>>>>> to attempt a migration. It further makes sense with our current work
>>>>>> towards upgrading Solr.
>>>>>>
>>>>>> We already have a battery of integration and manual tests after the
>>> ES
>>>>>> upgrade work that can be leveraged to validate the changes.
>>>>>>
>>>>>> Mike Miklavcic
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Jon
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> A.Nazemian
>>>
>>
>>
