Since enrichments, and even parsers, can be added on-the-fly, should the ES
indexer be intelligent enough to manage the index templates on-the-fly
also?  Ideally, I should never have to manually install something like an
ES template.  The indexer should just take care of all that.

In the case of the Elasticsearch indexer, if it notices a new field added
by an enrichment, or a new source of telemetry, then it should update the
ES template on-the-fly also.  Ideally, we would never have to manually
create/deploy an ES template.  It should all happen seamlessly and remain
in-sync with whatever enrichments, etc exist.





On Mon, Feb 20, 2017 at 2:36 PM, Nick Allen <[email protected]> wrote:

>
> Taking this a step further, I think this challenge goes beyond just
> parsers.  We would also need to solve this problem for enrichments.  When I
> add an enrichment, I want the enriched data to be indexed accurately.   How
> can we make that happen?
>
>    - As part of defining an enrichment, should I also be able to specify
>    the fields and types using this same generic definition?
>    - Or could this be inferred somehow via an extension to Stellar?
>
>
>
> On Mon, Feb 20, 2017 at 2:29 PM, Nick Allen <[email protected]> wrote:
>
>> I like the flexibility and extensibility of having some kind of internal
>> representation (generic definition) of the names and types of the fields
>> produced by a parser.
>>
>> Rather than shipping with an ES template, a parser would ship with a
>> generic definition of the field names and data types that it adds to a
>> message.  The Elasticsearch indexer would then take this generic definition
>> and translate it to an Elasticsearch template.
>>
>> Each new indexer (Solr, etc) would know how to consume this generic
>> definition and produce whatever artifacts that it needs to index the data
>> accurately.
>>
>>
>>
>>
>>
>>
>>
>> On Sat, Feb 18, 2017 at 12:26 PM, James Sirota <[email protected]>
>> wrote:
>>
>>> I am not sure I agree with packaging source-specific templates with the
>>> parser.  I think that would make it harder to add additional storage
>>> sources.  For example, what happens if I have 50 parsers with Solr and ES
>>> schemas defined, but now I want to add druid?  Now I have to add 50 schemas
>>> to all my  existing parsers, which I don't think makes sense.  I think what
>>> we should have instead is tuple mappers that map some internal
>>> representation of our schema to whatever schema the tool uses.  We already
>>> somewhat started to move down this path with Kyle defining the schema enum
>>> for his ASA parser PR and Simon defining a JSON schema for his CEF parser
>>> PR.  I think we need to unify these approaches and then propagate them to
>>> all the parsers.  I think what has to happen is the following:
>>>
>>> We have to introduce a partial schema for Metron messages where you can
>>> enforce a schema on a part of a message you want, but at the same time
>>> allow enough flexibility for the rest of the message to be flexible.  What
>>> I mean by that is that you should enforce a schema for things like ip,
>>> protocol, timestamp, etc, but have a fully flexible structure outside of
>>> that.
>>>
>>> After you do that then you can map the partial schema you defined to es,
>>> solr, druid, etc, etc.  For the fields you don't have a schema for you just
>>> assume they are strings.  To add additional storage/indexing source to
>>> Metron all you do is define a mapper to that source's schema and load that
>>> into our indexing bolt.
>>>
>>>
>>>
>>> Thanks,
>>> James
>>>
>>> 17.02.2017, 16:36, "[email protected]" <[email protected]>:
>>> > I think this is a good direction to move things toward - moving
>>> indexing
>>> > templates to be packaged with parsers (using multiple tiered options)
>>> that
>>> > are then merged with the possible enrich fields before getting added
>>> to the
>>> > indexing technology in use. Now, to read the proposal thread...
>>> >
>>> > Jon
>>> >
>>> > On Fri, Feb 17, 2017, 4:25 PM Simon Elliston Ball <
>>> > [email protected]> wrote:
>>> >
>>> >>  I’d broadly agree with that tiered approach.
>>> >>
>>> >>  The version where the parser emits a generic schema, and enrichments
>>> >>  contribute generic schema chunks to that which get combined into an
>>> indexer
>>> >>  specific template generated at the end of the flow, so yes, pretty
>>> much
>>> >>  inline with your proposal. (I did read though it, apologies if I
>>> missed any
>>> >>  of the detail, brain is still a little bit post-RSA!)
>>> >>
>>> >>  Simon
>>> >>
>>> >>  > On 17 Feb 2017, at 12:38, Otto Fowler <[email protected]>
>>> wrote:
>>> >>  >
>>> >>  > We already make them do this now, or they get the defaults. So
>>> this is
>>> >>  no different.
>>> >>  > Having parsers emit names and types etc, that would be another
>>> step - or
>>> >>  it could be the ‘generic schema’ as implemented actually.
>>> >>  >
>>> >>  > A tiered approach - from
>>> >>  > * you give nothing with the parser - you get whatever ES guesses
>>> at but
>>> >>  you don’t care do you
>>> >>  > * you give the schema
>>> >>  > * you give the types and we figure it out for you
>>> >>  >
>>> >>  > would be the best to move to.
>>> >>  >
>>> >>  > Also, we could use the names and types method tied to enrichment to
>>> >>  generate indexing templates for enrichment types or deriving them
>>> rather,
>>> >>  which i mention in my proposal.
>>> >>  >
>>> >>  > I’m starting to think you haven’t rushed out to read it Simon ;)
>>> >>  >
>>> >>  >
>>> >>  >
>>> >>  > On February 17, 2017 at 15:24:37, Simon Elliston Ball (
>>> >>  [email protected] <mailto:[email protected]>)
>>> wrote:
>>> >>  >
>>> >>  >> I like that, to an extent… Forcing the provision of explicit
>>> schema
>>> >>  might be a bit of a load for parser development. I’m assuming that
>>> custom
>>> >>  parsers would be pushed towards the same packaging approach.
>>> >>  >>
>>> >>  >> Would it make sense to require the parser to emit field names and
>>> types
>>> >>  expected, and then for us to provide a means of creating the
>>> templates for
>>> >>  supported indices, and push the actual template management to the
>>> index
>>> >>  layer rather than the parsing layer. Schema is after all determined
>>> not
>>> >>  just by a parser, but also by the combination of enrichments and
>>> models
>>> >>  applied.
>>> >>  >>
>>> >>  >> We could also of course provide an override option within your
>>> proposed
>>> >>  parser package model to allow any destination specific configuration
>>> of the
>>> >>  indexing template.
>>> >>  >>
>>> >>  >> Simon
>>> >>  >>
>>> >>  >> > On 17 Feb 2017, at 12:01, Otto Fowler <[email protected]
>>> >>  <mailto:[email protected]>> wrote:
>>> >>  >> >
>>> >>  >> > I think we can get there from my proposal.
>>> >>  >> > A source may package:
>>> >>  >> > * explicit schemas ( ES, SOLR, FOO )
>>> >>  >> > * a generic to be invented schema for a to be invented pluggable
>>> >>  indexing
>>> >>  >> > component :)
>>> >>  >> > and we’ll be able to handle it.
>>> >>  >> >
>>> >>  >> >
>>> >>  >> >
>>> >>  >> > On February 17, 2017 at 14:39:07, Kyle Richardson (
>>> >>  [email protected] <mailto:[email protected]>)
>>> >>  >> > wrote:
>>> >>  >> >
>>> >>  >> > I personally like the idea of a typed schema per parser that we
>>> could
>>> >>  >> > translate to multiple targets. This would allow us a lot more
>>> >>  modularity
>>> >>  >> > and extensibility in indexing down the road.
>>> >>  >> >
>>> >>  >> > -Kyle
>>> >>  >> >
>>> >>  >> > On Fri, Feb 17, 2017 at 1:59 PM, Simon Elliston Ball <
>>> >>  >> > [email protected] <mailto:[email protected]
>>> >>
>>> >>  wrote:
>>> >>  >> >
>>> >>  >> >> That sounds like a great idea Otto. Do you have any early
>>> design on
>>> >>  that
>>> >>  >> >> we can look at. Also, rather than just elastic templates do you
>>> >>  think we
>>> >>  >> >> should have some sort of typed schema we could translate to
>>> multiple
>>> >>  >> >> targets (solr, elastic, ur... other...) or are you thinking of
>>> >>  packaging
>>> >>  >> >> specific scheme assets like template json with the parser?
>>> >>  >> >>
>>> >>  >> >> Simon
>>> >>  >> >>
>>> >>  >> >>> On 17 Feb 2017, at 18:42, Otto Fowler <
>>> [email protected]
>>> >>  <mailto:[email protected]>> wrote:
>>> >>  >> >>>
>>> >>  >> >>>
>>> >>  >> >>> Not to jump the gun, but I’m crafting a proposal about
>>> parsers and
>>> >>  one
>>> >>  >> >> of the things I am going to propose relates to having the ES
>>> >>  Template for
>>> >>  >> > a
>>> >>  >> >> given parser installed or packaged with the parser. We could
>>> load the
>>> >>  >> >> template from there, edit, save and deploy etc. We can extend
>>> that
>>> >>  >> > concept
>>> >>  >> >> more and more later (drafts, versioning etc )
>>> >>  >> >>>
>>> >>  >> >>>
>>> >>  >> >>>> On February 17, 2017 at 13:22:45, Simon Elliston Ball (
>>> >>  >> >> [email protected] <mailto:simon@simonellistonbal
>>> l.com>)
>>> >>  wrote:
>>> >>  >> >>>>
>>> >>  >> >>>> A little while ago the issue of managing Elastic templates
>>> for new
>>> >>  >> >> sensor configs came up, and we didn’t quite put it to bed.
>>> >>  >> >>>>
>>> >>  >> >>>> When creating new sensors, I almost invariably find the
>>> >>  auto-generated
>>> >>  >> >> schemas for elastic pick some incorrect types. I also find I
>>> have to
>>> >>  >> >> recreate indexes every time to push in the proper dynamic
>>> templates
>>> >>  for
>>> >>  >> >> things like geo enrichment fields.
>>> >>  >> >>>>
>>> >>  >> >>>> So, my questions are:
>>> >>  >> >>>> How should we address elastic template for new sensors?
>>> >>  >> >>>> Do we have circumstances where we would need to configure
>>> types, or
>>> >>  >> > can
>>> >>  >> >> we get away with inferring them?
>>> >>  >> >>>> Should we just add some additional dynamic templates to
>>> cover our
>>> >>  >> >> common fields like timestamp (the most common culprit I find
>>> for
>>> >>  >> > incorrect
>>> >>  >> >> typing)?
>>> >>  >> >>>>
>>> >>  >> >>>> I’d also like to think about ways we can generalise this.
>>> Does
>>> >>  anyone
>>> >>  >> >> have any thoughts on what sort of additional index schemes we
>>> should
>>> >>  want
>>> >>  >> >> to infer (solr seems an obvious one, any others?).
>>> >>  >> >>>>
>>> >>  >> >>>> Thoughts on a well typed, schemaed and easily indexed
>>> postcard
>>> >>  please
>>> >>  >> > :)
>>> >>  >> >>>>
>>> >>  >> >>>> Simon
>>> >>  >> >>
>>> >>
>>> >>  --
>>> >
>>> > Jon
>>> >
>>> > Sent from my mobile device
>>>
>>> -------------------
>>> Thank you,
>>>
>>> James Sirota
>>> PPMC- Apache Metron (Incubating)
>>> jsirota AT apache DOT org
>>>
>>
>>
>

Reply via email to